Inference routing on agentgateway — KV-cache-aware routing to a self-hosted model pool

What you'll build

• One kind cluster with the Gateway API and Gateway API Inference Extension CRDs.
• agentgateway as the Gateway API data plane, with inferenceExtension.enabled=true.
• A self-hosted model pool: two llm-d inference simulator replicas behind one InferencePool.
• The GIE Endpoint Picker scoring replicas, and an InferenceObjective for serving priority.
• A Gateway and an HTTPRoute whose backend is the InferencePool, plus a notebook that drives the whole demo.

How the routing is wired

client ──/v1/chat/completions──▶ Gateway (agentgateway)
                                   │  HTTPRoute backendRef ─▶ InferencePool
                                   ▼                          (group inference.networking.k8s.io)
                             Endpoint Picker (EPP) ── scrapes ──▶ vllm:gpu_cache_usage_perc
                                   │                              vllm:num_requests_waiting
                       picks the replica with the best score
                         ┌─────────┴─────────┐
                    pool-a (cold)        pool-b (hot)

Step 1: cluster, Gateway API, and Inference Extension CRDs

Create the kind cluster and install the upstream Gateway API CRDs plus the GIE bundle, which brings the InferencePool (served as v1) and InferenceObjective kinds.

kind create cluster --config kind/inference.yaml

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

kubectl apply -f \
  https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.4.0/manifests.yaml

Step 2: install agentgateway with the Inference Extension on

inferenceExtension.enabled=true is what makes the controller watch InferencePool and delegate endpoint selection to the Endpoint Picker. The lab defaults to Solo Enterprise (needs a license); the OSS path is one env var away.

helm upgrade --install agentgateway-crds \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
  -n agentgateway-system --create-namespace --version v2.3.4

helm upgrade --install agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  -n agentgateway-system --version v2.3.4 \
  --set licensing.licenseKey="$AGENTGATEWAY_LICENSE_KEY" \
  --set inferenceExtension.enabled=true

helm upgrade --install agentgateway-crds \
  oci://cr.agentgateway.dev/charts/agentgateway-crds \
  -n agentgateway-system --create-namespace --version v1.3.1

helm upgrade --install agentgateway \
  oci://cr.agentgateway.dev/charts/agentgateway \
  -n agentgateway-system --version v1.3.1 \
  --set inferenceExtension.enabled=true

Step 3: the self-hosted model pool

Two model-server replicas, both labelled app: vllm-sim so the InferencePool selects them. Each mounts a config that pins its cache and queue gauges. Out of the box pool-a is cold (10% cache used, empty queue) and pool-b is hot (90% used, eight waiting). Those pinned values are what let you drive the routing decision on cue.

apiVersion: v1
kind: ConfigMap
metadata:
  name: sim-pool-a
  namespace: inference
data:
  config.yaml: |
    model: base-model
    port: 8000
    fake-metrics:
      kv-cache-usage: 0.10      # pool-a: cold, lots of free cache
      waiting-requests: 0
      running-requests: 1
# ... a matching Deployment runs ghcr.io/llm-d/llm-d-inference-sim:v0.5.0
#     with --config /etc/sim/config.yaml. pool-b is identical but kv-cache 0.90.

Step 4: InferencePool, Endpoint Picker, and the route

The GIE inferencepool Helm chart creates the InferencePool (selecting app=vllm-sim) and the Endpoint Picker that scores replicas. Then apply an InferenceObjective for priority, and a Gateway and HTTPRoute whose backend is the pool.

helm upgrade --install vllm-sim \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool \
  --version v1.4.0 -n inference \
  --set inferencePool.modelServers.matchLabels.app=vllm-sim \
  --set provider.name=none

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
  namespace: inference
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway
  rules:
    - backendRefs:
        - group: inference.networking.k8s.io   # the InferencePool, not a Service
          kind: InferencePool
          name: vllm-sim
      matches:
        - path: { type: PathPrefix, value: / }
      timeouts: { request: 300s }

Run it with the notebook

The whole demo is driven by demo.ipynb (a Bash-kernel notebook). Register the kernel once with ./demo-scripts/notebook-kernel.sh, stand the lab up, then step through the cells. Or drive it from the shell:

export SECRETS_FILE=/path/to/secrets-envs.sh   # exports AGENTGATEWAY_LICENSE_KEY
./scripts/quick.sh up
./scripts/quick.sh test        # fire requests, see which replica the picker chose

# OSS path, no license:
AGW_EDITION=oss ./scripts/quick.sh up

KV-cache-aware routing: flip it on cue

With pool-a cold and pool-b hot, every request lands on pool-a. Now saturate pool-a and free pool-b. Nothing about the route changes. The picker re-scores on the new metrics and traffic moves to pool-b, the replica that can actually serve fast.

./demo-scripts/route-test.sh 8         # 8/8 -> pool-a (cold)

./demo-scripts/set-kv.sh a 0.95 9 6    # pool-a now HOT
./demo-scripts/set-kv.sh b 0.05 0 1    # pool-b now COLD
sleep 6                                # let the picker re-scrape
./demo-scripts/route-test.sh 8         # 8/8 -> pool-b (traffic moved)

Serving priority with InferenceObjective

InferenceObjective attaches a priority to requests against the pool. Under contention the scheduler drains higher-priority work first, so interactive traffic keeps flowing while batch work is held back. Two tiers here: interactive (priority 10) and batch (priority 0). Priority bites when the pool is saturated, not when it is idle.

What the picker sees

The picker scores replicas from the vLLM metrics each server exposes. Read them straight off the pods. The cache-usage gauge and the waiting-queue gauge are the two that drive every decision above.

kubectl -n inference exec deploy/vllm-pool-a -- \
  sh -c 'wget -qO- localhost:8000/metrics' \
  | grep -E "gpu_cache_usage_perc|num_requests_waiting"
# vllm:gpu_cache_usage_perc{model_name="base-model"} 0.10
# vllm:num_requests_waiting{model_name="base-model"} 0

OSS and Enterprise

Every manifest in yaml/ is edition-neutral: it uses the shared agentgateway.dev and GIE APIs. The only per-edition difference is the GatewayClass name (injected at apply time) and the Helm chart, so the same yaml/ runs on both. That is why there is no separate yaml-oss/.

See also

• agentgateway — Inference routing docs
• Gateway API Inference Extension
• Sibling lab — vLLM Semantic Router on agentgateway as ExtProc

Versions

Built and verified on both editions: