Versioned Cluster Routing with Solo Enterprise for kgateway

What you'll build

All three are kind clusters on the shared docker network. The edge gateway reaches each app cluster as a static Backend whose host is the app cluster's node IP and whose port is the NodePort. Nothing routes by pod IP across clusters, so the app clusters stay completely independent of one another. The gateway is the only thing that knows the full set of versions.

Prerequisites

• kind, kubectl, helm, docker, openssl and jq on your path.
• gcloud, authenticated. The enterprise chart lives on a public Google Artifact Registry, but a helm OCI pull still needs a registry login with a gcloud access token. The script handles the login for you.
• A Solo Enterprise for kgateway license key. Export KGATEWAY_LICENSE_KEY, or point SECRETS_FILE at a sourceable file (the script also accepts GLOO_GATEWAY_LICENSE_KEY or SOLO_LICENSE_KEY).

Steps

Step 1 — Bring the whole thing up

One command creates the three clusters, installs kgateway, deploys the apps and wires the routing. It is idempotent, so a re-run just reconciles. The rest of the steps explain what each phase does.

# This lab needs ONE secret: a Solo Enterprise for kgateway license key.
# It is read from the first of these that is set:
#   KGATEWAY_LICENSE_KEY  (preferred)
#   GLOO_GATEWAY_LICENSE_KEY
#   SOLO_LICENSE_KEY

# Option A — export it in your shell:
export KGATEWAY_LICENSE_KEY="your-license-key"
./scripts/quick.sh up

# Option B — keep it in a sourceable file and point SECRETS_FILE at it.
# secrets.sh just needs the one line:
#   export KGATEWAY_LICENSE_KEY="your-license-key"
SECRETS_FILE=/path/to/secrets.sh ./scripts/quick.sh up

# individual phases, if you'd rather step through:
./scripts/01-clusters.sh    # 3 kind clusters + Gateway API CRDs on the edge
./scripts/02-kgateway.sh    # install enterprise-kgateway + the Gateway
./scripts/03-apps.sh        # deploy the version-stamped echo apps
./scripts/04-routing.sh     # Backends + JWKS + JWT policy + HTTPRoute

Step 2 — The three clusters

Three single-node kind clusters with distinct pod and service CIDRs so nothing collides on the shared docker network. The Gateway API CRDs (v1.5.1, the version kgateway 2.2 wants) go on the edge cluster only. The app clusters need nothing but a place to run the echo app.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: kgw-edge
networking:
  podSubnet: "10.30.0.0/16"
  serviceSubnet: "10.98.0.0/16"
nodes:
  - role: control-plane

Step 3 — Install Solo Enterprise for kgateway on the edge

Two charts from the public OCI registry: the CRDs chart, then the control plane. The license goes in as licensing.licenseKey. Creating the Gateway with gatewayClassName: enterprise-kgateway auto-provisions the Envoy proxy and the enterprise sidecars (ext-auth, ext-cache, rate-limiter, waf).

REG=oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts

helm upgrade --install enterprise-kgateway-crds \
  $REG/enterprise-kgateway-crds --version 2.2.0 \
  --namespace kgateway-system --create-namespace --wait

helm upgrade --install enterprise-kgateway \
  $REG/enterprise-kgateway --version 2.2.0 \
  --namespace kgateway-system \
  --set licensing.licenseKey="$KGATEWAY_LICENSE_KEY" --wait

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http
  namespace: kgateway-system
spec:
  gatewayClassName: enterprise-kgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

Step 4 — Deploy the versioned echo apps

The same echo app goes into both app clusters with a different CLUSTER_VERSION baked in. It is a stdlib-only Python server run straight off the stock python:3.12-alpine image with the script mounted from a ConfigMap, so there is no image to build. It answers any path with JSON that makes the routing outcome obvious: which cluster served the request, and the request headers the gateway forwarded. It is exposed on NodePort 30080.

Step 5 — Wire the Backends, the JWT policy and the route

Two static Backends, one per versioned cluster. The host and port are the app cluster's node IP and NodePort, discovered and filled in at setup time.

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: app-latest
  namespace: kgateway-system
spec:
  type: Static
  static:
    hosts:
      - host: 192.168.97.15   # app-latest node IP
        port: 30080           # NodePort
---
apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: app-v2
  namespace: kgateway-system
spec:
  type: Static
  static:
    hosts:
      - host: 192.168.97.16   # app-v2 node IP
        port: 30080

The JWT policy validates the token against an inline JWKS and maps the version claim into x-target-version. validationPolicy: AllowMissing lets tokenless requests through, so the override and default paths keep working, while a present but invalid token is still rejected. The JWKS is inline (jwks.local.key), so the whole lab runs offline with no JWKS server, and mint-token.sh signs matching RS256 tokens.

apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayTrafficPolicy
metadata:
  name: jwt-version
  namespace: kgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: http
  entJWT:
    beforeExtAuth:
      validationPolicy: AllowMissing
      providers:
        tenant-idp:
          issuer: versioned-routing-lab
          audiences:
            - public-api
          tokenSource:
            headers:
              - header: authorization
                prefix: "Bearer "
          jwks:
            local:
              key: |
                { "keys": [ { "kty": "RSA", "use": "sig", "alg": "RS256",
                              "kid": "versioned-routing-key", "n": "...", "e": "AQAB" } ] }
          claimsToHeaders:
            - claim: version
              header: x-target-version

The route matches an explicit override header first, then the JWT-derived header, then falls through to latest. Gateway API ranks a rule with a header match above the bare path rule, and breaks ties by rule order, so this gives override > claim > default. The two version inputs sit on two different headers on purpose, which the demo explains.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: versioned-routing
  namespace: kgateway-system
spec:
  parentRefs:
    - name: http
      namespace: kgateway-system
  rules:
    # 1. explicit client/ops override (not touched by the JWT filter)
    - matches:
        - headers:
            - name: x-version-override
              value: v2
              type: Exact
      backendRefs:
        - group: gateway.kgateway.dev
          kind: Backend
          name: app-v2
    # 2. JWT-claim-derived version
    - matches:
        - headers:
            - name: x-target-version
              value: v2
              type: Exact
      backendRefs:
        - group: gateway.kgateway.dev
          kind: Backend
          name: app-v2
    # 3. default to latest
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - group: gateway.kgateway.dev
          kind: Backend
          name: app-latest

Run the demo

./scripts/quick.sh demo port-forwards the gateway proxy to localhost:8080 and runs seven requests. The echo app reports which cluster served each one and which headers it saw, so every row below is observed against the running gateway, not assumed.

kubectl --context kind-kgw-edge -n kgateway-system port-forward svc/http 8080:80 &

TOKEN=$(./scripts/mint-token.sh v2 acme)     # version=v2, tenant=acme
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/ | jq .servedBy
# "v2"

curl -s -H "x-version-override: v2" http://localhost:8080/ | jq .servedBy
# "v2"

The JWT-claim path also confirms something worth knowing about kgateway: claimsToHeaders sets the header early enough that the HTTPRoute re-evaluates and routes on it, and the same header reaches the backend. So a version claim is a real routing input, not just metadata for the upstream.

Keeping the tenant-to-version map current

Keep the map in the IdP as a claim, keyed off the tenant. A version rollover is then an identity change: update the claim the IdP issues for that tenant, and the gateway config does not change. There is no separate lookup table to keep in sync on the gateway, and no header for a tenant to spoof. Clients that have no version claim, plus ops doing a forced test, use the explicit x-version-override header instead.

Issuing the version claim from a real IdP — Frontegg

The lab signs its own RS256 tokens against an inline JWKS so it runs offline. In production the version claim comes from your IdP, keyed off the user or tenant. Here is how to issue it from Frontegg, a hosted CIAM provider, so the same claimsToHeaders policy routes on a real identity. Sample environment host throughout: acme.frontegg.com.

1. Set the per-user version as user metadata. Frontegg carries arbitrary per-user data in a metadata field and surfaces it in the access token as a userMetadata claim. Set it from the portal (Users → the user → Metadata) or with the management API:

# VTOK = a vendor (environment) JWT from POST https://api.frontegg.com/auth/vendor/
curl -s -X PUT "https://api.frontegg.com/identity/resources/users/v1/$ALICE_USER_ID" \
  -H "Authorization: Bearer $VTOK" \
  -H 'Content-Type: application/json' \
  -d '{"metadata":"{\"version\":\"v2\"}"}'

A token minted for that user now carries the value under userMetadata (this is a real decoded Frontegg token, host redacted to the sample):

{
  "sub": "<client-id>",
  "name": "Alice",
  "email": "alice@acme.io",
  "userMetadata": { "version": "v2" },
  "tenantId": "acme",
  "roles": ["Admin"],
  "userId": "<alice-user-id>",
  "iss": "https://acme.frontegg.com",
  "aud": "<environment-id>"
}

2. Read the nested claim directly. Frontegg nests the value under userMetadata, and nested claims are supported: claimsToHeaders maps the claim name straight onto Envoy's claim_to_headers, which resolves a dot-separated path into a nested object. So point the claim at userMetadata.version and the gateway pulls out v2 on its own:

claimsToHeaders:
  - claim: userMetadata.version   # dot-path into the nested object
    header: x-target-version

The dot path resolves to v2, which lands in x-target-version — the same header the route already matches, so the routing table doesn't change. (Envoy base64-encodes a claim only when the value is itself an object; a leaf string like version is copied verbatim.)

3. Point the JWT policy at Frontegg instead of the inline key. This is the same yaml/edge/jwt-policy.yaml from Step 5, with the provider swapped to Frontegg: a real issuer, a remote JWKS instead of the inline key, and the nested claim (confirm issuer and JWKS from Authentication → SSO → Identity Provider → OpenID Connect Endpoints). Apply it to the edge cluster in place of the lab's self-signed version:

apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayTrafficPolicy
metadata:
  name: jwt-version
  namespace: kgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: http
  entJWT:
    beforeExtAuth:
      validationPolicy: AllowMissing
      providers:
        tenant-idp:
          issuer: https://acme.frontegg.com
          audiences:
            - <environment-id>        # Frontegg aud = the environment / client id
          tokenSource:
            headers:
              - header: authorization
                prefix: "Bearer "
          jwks:
            remote:
              url: https://acme.frontegg.com/.well-known/jwks.json
          claimsToHeaders:
            - claim: userMetadata.version
              header: x-target-version

Teardown

./scripts/quick.sh teardown    # deletes kgw-edge, app-latest and app-v2

See also

• Part 2 — the same lab on Solo Enterprise for agentgateway
• Solo Enterprise for kgateway docs
• kgateway vs Istio ingress — picking a north-south gateway
• Per-team LLM token budgets at the gateway — JWT claims to headers, the agentgateway sibling

Versions

Built and verified on: