Intelligent LLM routing with the vLLM Semantic Router on agentgateway

Create a local cluster (optional)

Skip if you already have a cluster.

kind create cluster --name semantic-router-agentgateway
kubectl wait --for=condition=Ready nodes --all --timeout=300s

Install agentgateway

Install the Gateway API CRDs, then the agentgateway CRDs and controller. Experimental Gateway API features are enabled because ExtProc rides on them.

export AGENTGATEWAY_VERSION=v1.3.0-alpha.1

kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --create-namespace \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  --set controller.image.pullPolicy=Always

helm upgrade -i agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  --set controller.image.pullPolicy=Always \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true \
  --wait

kubectl get pods -n agentgateway-system

Create the gateway proxy

A standard Gateway API Gateway on the agentgateway class, listening on port 80.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: All

kubectl apply -f gateway.yaml
kubectl wait --for=condition=Available deployment/agentgateway-proxy \
  -n agentgateway-system --timeout=300s

Deploy a demo vLLM-compatible backend

An OpenAI-compatible simulator serving a base model plus six LoRA adapters (math, science, social, humanities, law, general) — the specialists the router will route to. The full Deployment + Service is in the docs; the shape is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b-instruct
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm-llama3-8b-instruct }
  template:
    metadata:
      labels: { app: vllm-llama3-8b-instruct }
    spec:
      containers:
      - name: vllm-sim
        image: ghcr.io/llm-d/llm-d-inference-sim:v0.5.0
        args:
        - --model
        - base-model
        - --port
        - "8000"
        - --max-loras
        - "6"
        - --lora-modules
        - '{"name": "math-expert"}'
        - '{"name": "science-expert"}'
        - '{"name": "social-expert"}'
        - '{"name": "humanities-expert"}'
        - '{"name": "law-expert"}'
        - '{"name": "general-expert"}'
        ports:
        - { containerPort: 8000, name: http }
        readinessProbe:
          httpGet: { path: /health, port: http }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama3-8b-instruct
  namespace: default
spec:
  selector: { app: vllm-llama3-8b-instruct }
  ports:
  - { port: 8000, targetPort: 8000 }

kubectl wait --for=condition=Available deployment/vllm-llama3-8b-instruct \
  -n default --timeout=300s

Deploy the Semantic Router

Install the router via Helm with the agentgateway-specific values file, which points it at the demo backend and configures the LoRA adapter selection.

helm install semantic-router oci://ghcr.io/vllm-project/charts/semantic-router \
  --version v0.0.0-latest \
  --namespace agentgateway-system \
  -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/agentgateway/semantic-router-values/values.yaml

kubectl wait --for=condition=Available deployment/semantic-router \
  -n agentgateway-system --timeout=600s

Create the routing resources

An AgentgatewayBackend pointing at the vLLM service, and an HTTPRoute binding it to the gateway. Note the backend omits openai.model on purpose — so the model name the router writes into the request body is the one that's used.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: semantic-router-vllm
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai: {}          # model intentionally omitted
      host: vllm-llama3-8b-instruct.default.svc.cluster.local
      port: 8000
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: semantic-router-vllm
  namespace: agentgateway-system
spec:
  parentRefs:
  - name: agentgateway-proxy
    namespace: agentgateway-system
  rules:
  - backendRefs:
    - name: semantic-router-vllm
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: semantic-router-extproc
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
  traffic:
    extProc:
      backendRef:
        name: semantic-router
        namespace: agentgateway-system
        port: 50051
      processingOptions:
        requestHeaderMode: Send
        requestBodyMode: Buffered
        responseHeaderMode: Send
        responseBodyMode: Buffered
        allowModeOverride: true

Send a request through it

Port-forward the gateway and send an OpenAI-style request with "model": "auto". The router classifies the maths prompt, selects the maths route, and rewrites the request before agentgateway forwards it — the client never names a concrete model.

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80

curl -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "What is the derivative of f(x) = x^3?"}
    ],
    "max_tokens": 64,
    "temperature": 0
  }'

Troubleshooting

If a request doesn't route as expected, walk the path from the gateway to the router to the backend.

# Gateway accepted and programmed?
kubectl get gateway agentgateway-proxy -n agentgateway-system
kubectl logs -n agentgateway-system deployment/agentgateway

# Routing resources accepted?
kubectl describe httproute semantic-router-vllm -n agentgateway-system
kubectl describe agentgatewaybackend semantic-router-vllm -n agentgateway-system

# ExtProc wired and the router healthy?
kubectl get svc semantic-router -n agentgateway-system
kubectl logs -n agentgateway-system deployment/semantic-router
kubectl describe agentgatewaypolicy semantic-router-extproc -n agentgateway-system

# Demo backend serving?
kubectl get pods -n default -l app=vllm-llama3-8b-instruct
kubectl logs -n default deployment/vllm-llama3-8b-instruct

Takeaways for production

Working with LoRA adapters

This guide leans on LoRA adapters throughout, so here is the practical detail in one place: what they are, where to get them, how to load them, and how the router consumes them. One framing first — adapters are a backend (vLLM) concern. agentgateway and the Semantic Router only ever pass a model-name string; the adapters themselves live entirely in the serving layer behind the AgentgatewayBackend.

What a LoRA adapter is

LoRA (Low-Rank Adaptation) is a lightweight fine-tuning technique. Rather than train a whole separate model per domain, you train a small set of extra weights — the adapter — that layers on top of one shared base model. The base model is loaded into GPU memory once; many adapters attach to it and can be swapped per request. That's how a single vLLM backend serves a base model plus several domain “experts” without the cost of several full models. Adapters are small — megabytes, not gigabytes.

Where to get the adapters

• Train your own. LoRA fine-tune the base model on your domain data (maths, legal, support transcripts, …). The output is a small directory containing adapter_config.json and adapter_model.safetensors.
• Download pre-trained ones. Hubs like Hugging Face host ready-made LoRA adapters. The hard requirement either way: an adapter must be trained against the same base model you're serving, or vLLM will refuse to load it.

How to load them in vLLM

Serve the base model with LoRA enabled, then register each adapter either statically at startup or dynamically at runtime.

Static — declared on the serve command. Each entry is name=path, where the path is a local directory or a model-hub repo:

vllm serve base-model \
  --enable-lora \
  --max-loras 6 --max-lora-rank 16 \
  --lora-modules \
    math-expert=/models/math-lora \
    law-expert=/models/law-lora \
    science-expert=/models/science-lora

Dynamic — load and unload without a restart. Set VLLM_ALLOW_RUNTIME_LORA_UPDATING=true on the server, then call the admin endpoints:

curl -X POST http://$BACKEND:8000/v1/load_lora_adapter \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "math-expert", "lora_path": "/models/math-lora"}'

curl -X POST http://$BACKEND:8000/v1/unload_lora_adapter \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "math-expert"}'

How to discover what's loaded

vLLM is OpenAI-compatible, so GET /v1/models lists the base model and every loaded adapter as its own entry (each adapter references the base model as its parent). This is the canonical answer to “what can I route to right now” — for an operator, a client, or for sanity-checking the router's config:

curl http://$BACKEND:8000/v1/models
# → base-model, math-expert, law-expert, science-expert, …

How they're consumed here

Once loaded, an adapter is addressed exactly like a model: put "model": "math-expert" in the request body and vLLM applies that adapter on top of the base model for the request. In this architecture the client never does that — it sends "model": "auto", and the Semantic Router writes the chosen adapter name into the body before agentgateway forwards it. So consuming the adapter is the router's job; your job is to load the adapters in vLLM and map each domain category to its adapter name in the router's configuration.

The full, always-current manifests live in the official guide: vllm-semantic-router.com/docs/installation/k8s/agentgateway. For where this sits alongside identity, authorization, and governance for agentic traffic, see Eight enterprise controls for MCP traffic in agentgateway policy and Securing MCP and agentic systems.