Ambient Deployment and L4/L7 Policies Demo

What you'll build

A petshop in namespace petshop, enrolled in ambient. One API and four clients, each with its own identity:

Trust domain is the cluster name, so identities are spiffe://cert-identity/ns/petshop/sa/<sa> — not cluster.local. Keycloak runs in its own non-ambient namespace with users alice (role user) and bob (role admin).

Stand up the mesh

There are two ways to install Solo Istio: the Gloo Operator, which manages the mesh for you, or plain Helm, which makes you spell out every setting yourself. This lab uses plain Helm — four charts (base, istiod, cni, ztunnel), profile ambient — because three settings decide how the whole lab behaves, and you should see exactly where they come from:

• the trust domain (meshConfig.trustDomain: cert-identity) — this names every identity in the mesh. Each policy on this page matches on cert-identity/ns/petshop/sa/…; get this wrong (or assume the default cluster.local) and every policy silently matches nothing.
• the licence (license.value on istiod) — unlocks the Solo pieces used later: the enriched telemetry, the workload claims, the agentgateway waypoint.
• JSON access logs (LOG_FORMAT: json on ztunnel) — the evidence trail. The allow/deny verdicts you'll read in the access-log step come out structured because of this one value.

All three are set as Helm values at install time. With the operator you would set the same things through its CRD; either way, nothing in this lab is patched into place afterwards — what you see below is the complete configuration.

make setup SECRETS_FILE=~/code/solo/secrets/secrets-envs.sh

HUB=us-docker.pkg.dev/soloio-img/istio ; TAG=1.30.3-solo   # image tag KEEPS -solo on the 1.30 line
HREPO=oci://us-docker.pkg.dev/soloio-img/istio-helm         # chart repo
HVER=1.30.3-solo                                            # chart version (same)

helm upgrade -i istio-base $HREPO/base   -n istio-system --create-namespace --version $HVER --set defaultRevision=default

helm upgrade -i istiod $HREPO/istiod -n istio-system --version $HVER -f - <<EOF
profile: ambient
global: { hub: $HUB, tag: $TAG }
istio_cni: { enabled: true }
license: { value: $SOLO_ISTIO_LICENSE_KEY }   # licence as a value
meshConfig:
  accessLogFile: /dev/stdout
  trustDomain: cert-identity                  # the trust domain, set directly
EOF

helm upgrade -i istio-cni $HREPO/cni -n istio-system --version $HVER -f - <<EOF
profile: ambient
global: { hub: $HUB, tag: $TAG }
EOF

helm upgrade -i ztunnel $HREPO/ztunnel -n istio-system --version $HVER -f - <<EOF
profile: ambient
hub: $HUB
tag: $TAG
env: { LOG_FORMAT: json, L7_ENABLED: "true" }  # JSON logs + Solo L7 telemetry
EOF

Watch it in the Solo UI (Gloo UI)

Every policy in this lab has a visual counterpart: workloads appearing, traffic flowing, edges going quiet when a policy blocks them. The Gloo UI is where you watch that happen — Solo's dashboard for the mesh, backed by the Gloo Platform management plane (on one kind cluster the management server, the agent and the UI all run together; registering the cluster is what lets the agent discover the mesh). This step is optional, but the Graph moments called out later on this page assume you have it. Needs GLOO_PLATFORM_LICENSE_KEY (falls back to SOLO_ISTIO_LICENSE_KEY).

It has no Istio dependency and is the slowest install in the lab, so the notebook kicks it off in the background the moment the kind cluster exists and just checks the result here. Install it before the petshop either way, to watch the workloads appear.

helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
  -n gloo-mesh --create-namespace --version 2.13.2 --wait

# register BEFORE the main install — else the agent crashloops "not registered"
# and a helm --wait hangs on it for the full timeout
kubectl apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata: { name: cert-identity, namespace: gloo-mesh }
spec: { clusterDomain: cluster.local }
EOF

# main install WITHOUT --wait (it blocks until every pod is Ready); wait only for the UI
helm upgrade -i gloo-platform gloo-platform/gloo-platform \
  -n gloo-mesh --version 2.13.2 -f - <<EOF
common: { cluster: cert-identity }
licensing: { glooMeshLicenseKey: "$GLOO_PLATFORM_LICENSE_KEY" }
glooMgmtServer: { enabled: true, createGlobalWorkspace: true }
glooUi: { enabled: true, serviceType: ClusterIP }
glooAgent: { enabled: true, relay: { serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900 } }
prometheus: { enabled: true }
redis: { deployment: { enabled: true } }
telemetryCollector: { enabled: true }
telemetryGateway: { enabled: true }
EOF
kubectl -n gloo-mesh rollout status deploy/gloo-mesh-ui --timeout=300s

# 8090 is often taken by another lab, so bind a free random local port
PORT=$(python3 -c 'import socket;s=socket.socket();s.bind(("",0));print(s.getsockname()[1]);s.close()')
nohup kubectl -n gloo-mesh port-forward svc/gloo-mesh-ui $PORT:8090 \
  > /tmp/cert-identity-gloo-ui-pf.log 2>&1 &
echo "Gloo UI → http://localhost:$PORT"

Deploy the petshop

Not the stock Istio sample — five tiny workloads built for this lab. petstore is an inline Python API (GET /pets, DELETE /pets/{id}); the other four are clients that curl it in a loop every couple of seconds, so every policy you apply on this page shows its effect within seconds in the client logs, the ztunnel access logs and the Graph — no traffic generator to set up. The ServiceAccount layout is the point: storefront and analytics each get their own SA, while the two checkout pods deliberately share sa/checkout to set up the shared-identity problem this lab closes later.

kubectl label ns petshop istio.io/dataplane-mode=ambient   # (created by the manifest)
kubectl apply -f yaml/10-app/        # petstore + storefront + analytics + checkout(blue/green)

Identity is the certificate

ztunnel holds one mTLS SVID per workload identity, and the identity is the ServiceAccount — the URI SAN is exactly what an L4 policy matches on. So look for what is missing below: five pods, but only four leaf certs. checkout-blue and checkout-green never appear by name; they share sa/checkout, so ztunnel presents one cert for both. That is the ceiling the shared-SA step hits, and the workload-claims step removes.

ZT=$(kubectl -n istio-system get pod -l app=ztunnel \
      --field-selector spec.nodeName=cert-identity-worker -o jsonpath='{.items[0].metadata.name}')
istioctl ztunnel-config certificate "$ZT.istio-system" | grep petshop

spiffe://cert-identity/ns/petshop/sa/analytics    Leaf   Available   true
spiffe://cert-identity/ns/petshop/sa/checkout     Leaf   Available   true    <- shared by blue AND green
spiffe://cert-identity/ns/petshop/sa/petstore     Leaf   Available   true
spiffe://cert-identity/ns/petshop/sa/storefront   Leaf   Available   true

Authorize on identity, at L4

One AuthorizationPolicy, enforced by ztunnel. It selects petstore and allows only the storefront identity. ztunnel fails closed, so this one policy allows storefront and denies everyone else — no app change, no waypoint.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: allow-storefront, namespace: petshop }
spec:
  selector: { matchLabels: { app: petstore } }
  action: ALLOW
  rules:
  - from: [{ source: { principals: ["cert-identity/ns/petshop/sa/storefront"] } }]
    to:   [{ operation: { ports: ["8080"] } }]

storefront:     GET petstore/pets : 200
analytics:      GET petstore/pets : 000000     <- denied
checkout-blue:  GET petstore/pets : 000000     <- denied
checkout-green: GET petstore/pets : 000000     <- denied

Read the L4 access logs

ztunnel logs every connection with the peer SPIFFE identities and the outcome — identity-aware audit at L4.

{"src.identity":"spiffe://cert-identity/ns/petshop/sa/analytics", "dst.service":"petstore.petshop...",
 "error":"connection closed due to policy rejection: allow policies exist, but none allowed"}

{"scope":"http_access","src.identity":"spiffe://cert-identity/ns/petshop/sa/storefront",
 "method":"GET","path":"/pets","response_code":200}

The shared-ServiceAccount ceiling

Add sa/checkout to the allowed set. Both checkout pods get in — and there is no L4 rule that lets one in and keeps the other out. The identity is the certificate, the certificate is issued per ServiceAccount, and both pods present the same sa/checkout SVID. To ztunnel they are literally the same caller.

Blue/green makes the problem easy to see, but the everyday version is worse: any pod that never sets serviceAccountName runs as the namespace's default ServiceAccount. In a namespace where nobody created dedicated SAs, every pod shares sa/default — one identity for the lot. Write an L4 ALLOW for "the payments app" and you have admitted every workload in that namespace, including the debug pod someone kubectl run'd last week. The policy looks precise; the identity underneath it is not.

Two fixes, and this lab shows the second: give every workload its own ServiceAccount as a baseline (as this app does), and where pods legitimately share one — blue/green deployments, horizontally-split variants of the same service — close the gap with workload claims.

kubectl apply -f yaml/20-policy/20-allow-checkout.yaml   # adds principal sa/checkout

What else can L4 decide? Namespace, conditions, DENY

Identity is the headline, but ztunnel authorizes on the whole L4 tuple: source identity, source namespace, source IP, destination port and connection SNI — directly or in a CEL when clause — and a DENY beats any ALLOW. All in ztunnel, no waypoint. To decide on where a caller is, we add a caller in a second namespace, warehouse, and walk an arc you can follow in the Gloo UI Graph: allow it and watch it appear, then block it and watch it stop. Drop the Graph's time-interval picker to the smallest window first — the graph draws served requests over that window, so a small window makes both halves visible within a minute or two.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: l4-allow-petshop-namespace, namespace: petshop }
spec:
  selector: { matchLabels: { app: petstore } }
  action: ALLOW
  rules:
  - to:   [{ operation: { ports: ["8080"] } }]
    when: [{ key: source.namespace, values: ["petshop", "warehouse"] }]

petshop:   storefront -> GET petstore/pets : 200
warehouse: warehouse-svc -> GET petstore.petshop/pets : 200   <- in the when list, so in

spec:
  selector: { matchLabels: { app: petstore } }
  action: ALLOW
  rules:
  - to:   [{ operation: { ports: ["8080"] } }]
    when: [{ key: source.namespace, values: ["petshop"] }]

From the client, both denials are the same 000000 — the difference is in ztunnel's access log, which names the mechanism (and for an explicit DENY, the policy):

spiffe://cert-identity/ns/warehouse/sa/warehouse-svc -> petstore.petshop.svc.cluster.local
  error: connection closed due to policy rejection: allow policies exist, but none allowed

spiffe://cert-identity/ns/petshop/sa/analytics -> petstore.petshop.svc.cluster.local
  error: connection closed due to policy rejection: explicitly denied by: petshop/l4-deny-analytics

Closing the shared-ServiceAccount gap — workload claims

The shared-ServiceAccount step hit a wall: two pods on one ServiceAccount are indistinguishable at L4. Workload claims close it: with ENABLE_WORKLOAD_CLAIMS=true, ztunnel requests a certificate per pod, istiod embeds claims in it at issuance, and the policy matches them with CEL — still at L4, still no waypoint. The mesh is already on 1.30.3-solo, so this is one Helm value on ztunnel. The flag stayed off until now on purpose — the shared-cert gap is the story this step closes. It is pure L4: no waypoint exists yet, and none is needed.

helm upgrade -i ztunnel $HREPO/ztunnel -n istio-system --version $HVER -f - <<EOF
profile: ambient
hub: $HUB
tag: $TAG
namespace: istio-system
istioNamespace: istio-system
env:
  LOG_FORMAT: json
  L7_ENABLED: "true"
  ENABLE_WORKLOAD_CLAIMS: "true"    # per-POD certs + claim enforcement
EOF

CERTIFICATE NAME                                                                                      TYPE  STATUS
spiffe://cert-identity/ns/petshop/sa/checkout@Kubernetes//Pod/petshop/checkout-blue-78f4c64c9d-jdv95  Leaf  Available (WIT present)
spiffe://cert-identity/ns/petshop/sa/checkout@Kubernetes//Pod/petshop/checkout-green-c68b9699f-mnxjj  Leaf  Available (WIT present)

Annotate the pod — the claim is embedded in its certificate at issuance, alongside auto claims for the workload name, namespace and pod. The SPIFFE URI never changes (still sa/checkout); the claims ride alongside it. Open the cert with openssl and the claim is right there, signed by istiod:

kubectl -n petshop patch deploy checkout-blue  -p '{"spec":{"template":{"metadata":{"annotations":{"solo.io.security-claims/tier":"gold"}}}}}'
kubectl -n petshop patch deploy checkout-green -p '{"spec":{"template":{"metadata":{"annotations":{"solo.io.security-claims/tier":"silver"}}}}}'

# grab blue's leaf cert from ztunnel (istioctl ztunnel-config certificates -o json
# base64-wraps the PEM), then open it: the new otherName SAN under Solo's OID
# carries the claims as base64url JSON (the notebook decodes it inline)
openssl x509 -in checkout-blue.pem -noout -text | grep -A1 "Subject Alternative Name"
#   URI:spiffe://cert-identity/ns/petshop/sa/checkout, othername: 1.3.6.1.4.1.65865.1.1:eyJpc3MiOi…

{
  "iss": "https://istiod.istio-system.svc.cert-identity",
  "sub": "spiffe://cert-identity/ns/petshop/sa/checkout",
  "istio.io": { "trust_domain": "cert-identity",
                "workload": { "name": "checkout", "namespace": "petshop", "pod": "checkout-blue-78f4c64c9d-jdv95" } },
  "solo.io":  { "security-claims": { "tier": "gold" } }
}

Authorize on it with the same when CEL shape as the namespace step, over source.claims (the / in the annotation key becomes . in the claim key):

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-gold-checkout
  namespace: petshop
spec:
  selector:
    matchLabels: { app: petstore }
  action: ALLOW
  rules:
    - from: [{ source: { principals: ["cert-identity/ns/petshop/sa/checkout"] } }]
      to:   [{ operation: { ports: ["8080"] } }]
      when:
        - key: "source.claims['solo.io.security-claims.tier']"
          values: ["gold"]

attached to ztunnel          <- the policy status: ztunnel itself enforces this
checkout-blue  -> 200
checkout-green -> DENIED at L4

Add a waypoint for L7 — and the waypoint is agentgateway

Everything so far was L4 — including the workload claims. HTTP concerns — methods, paths, JWTs, rate limits — need a waypoint, and on Solo Enterprise the waypoint data plane is agentgateway (GatewayClass enterprise-agentgateway-waypoint), not Envoy. The division of labour stays clean: ztunnel keeps proving the caller's SPIFFE identity over HBONE at L4, then hands the connection to agentgateway for L7 — and the workload identity ztunnel proved rides along, available to every policy as source.identity.*. You opt in at the namespace level (one waypoint for every service in petshop, the common default) or per-service. We reset the L4 policies here so the L7 story stands on its own (in production you keep both — defence in depth).

One shared-CRD note before the install: gloo-platform-crds (the Gloo UI step) and enterprise-agentgateway-crds both ship authconfigs and ratelimitconfigs, and Helm refuses to adopt CRDs owned by another release — hand those two to the agentgateway release first.

# the two CRDs shared with gloo-platform-crds move to the agentgateway release first
for crd in authconfigs.extauth.solo.io ratelimitconfigs.ratelimit.solo.io; do
  kubectl annotate crd $crd meta.helm.sh/release-name=agentgateway-crds \
    meta.helm.sh/release-namespace=agentgateway-system --overwrite
done
helm upgrade -i agentgateway-crds \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
  -n agentgateway-system --create-namespace --version v2026.7.0 --wait
# clusterName must match the mesh cluster id (ISTIO_META_CLUSTER_ID = Kubernetes here)
helm upgrade -i agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  -n agentgateway-system --version v2026.7.0 \
  --set licensing.licenseKey=$AGENTGATEWAY_LICENSE_KEY --set clusterName=Kubernetes --wait

# the waypoint: a Gateway of class enterprise-agentgateway-waypoint
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: petshop-waypoint, namespace: petshop, labels: { istio.io/waypoint-for: service } }
spec:
  gatewayClassName: enterprise-agentgateway-waypoint
  listeners: [{ name: mesh, port: 15088, protocol: HTTP }]
---
# namespace-scoped: one waypoint for every service in petshop
kubectl label namespace petshop istio.io/use-waypoint=petshop-waypoint

Deploy an IdP and mint a JWT

Keycloak runs in its own non-ambient namespace with a petshop realm and two users. We mint a token with the password grant and read its claims — the issuer and realm_access.roles are what the waypoint will check.

KC=http://keycloak.keycloak.svc.cluster.local:8080/realms/petshop/protocol/openid-connect/token
kubectl -n petshop exec deploy/storefront -- sh -c \
  "curl -s -d grant_type=password -d client_id=petshop -d username=bob -d password=bob -d scope=openid $KC"
# then base64-decode the payload segment

iss:   http://keycloak.keycloak.svc.cluster.local:8080/realms/petshop
user:  bob
roles: ['admin', 'user']

Authorize on the JWT, at L7

Two EnterpriseAgentgatewayPolicy objects on the waypoint. jwtAuthentication (mode: Strict) validates every request's token against Keycloak's JWKS, fetched in-cluster via the backendRef — no token or a bad token is 401. authorization then decides with CEL over the validated claims: any valid token may GET; DELETE additionally requires realm_access.roles to contain admin. The matchExpressions are OR'd — a request passes if any one holds.

kind: EnterpriseAgentgatewayPolicy   # authenticate: Strict JWT against Keycloak's JWKS
spec:
  targetRefs: [{ group: gateway.networking.k8s.io, kind: Gateway, name: petshop-waypoint }]
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
        - issuer: http://keycloak.keycloak.svc.cluster.local:8080/realms/petshop
          jwks:
            remote:
              backendRef: { name: keycloak, namespace: keycloak, port: 8080 }
              jwksPath: /realms/petshop/protocol/openid-connect/certs
---
kind: EnterpriseAgentgatewayPolicy   # authorize: CEL over the validated claims
spec:
  targetRefs: [{ group: gateway.networking.k8s.io, kind: Gateway, name: petshop-waypoint }]
  traffic:
    authorization:
      action: Allow
      policy:
        matchExpressions:
          - 'request.method == "GET"'
          - 'request.method == "DELETE" && "admin" in jwt.realm_access.roles'

no token    GET  /pets     -> 401     <- authentication (Strict JWT)
alice       GET  /pets     -> 200
alice(user) DELETE /pets/1 -> 403     <- authorization (CEL)
bob(admin)  DELETE /pets/1 -> 200

Route at the waypoint — canary and header shift

The waypoint is not just a policy point — agentgateway is a standard Gateway API data plane, so the proxy that checks the JWT also routes. An HTTPRoute whose parentRef is the petstore Service (the GAMMA pattern) gives callers a 90/10 canary to a new petstore-v2 and a header shift (x-beta: true goes straight to v2) — callers keep calling petstore:8080, and identity does not change (v2 runs the same ServiceAccount). The JWT policy still gates every request at the same waypoint, whichever version it lands on.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: petstore-split, namespace: petshop }
spec:
  parentRefs:
    - group: ""
      kind: Service        # mesh routing: the parent is the SERVICE, no ingress
      name: petstore
      port: 8080
  rules:
    - matches:
        - headers: [{ name: x-beta, value: "true" }]
      backendRefs:
        - { name: petstore-v2, port: 8080 }
    - backendRefs:
        - { name: petstore,    port: 8080, weight: 90 }
        - { name: petstore-v2, port: 8080, weight: 10 }

18 petstore          <- the 90%
 2 petstore-v2       <- the canary 10%

 5 petstore-v2       <- x-beta: true, every time

no token -> 401      <- the JWT policy still applies on the beta route

Rate limit — by workload identity

The user's token says who the user is; the certificate says who the workload is. Here both meet: a rate limit whose CEL condition keys on source.identity.serviceAccount — the SPIFFE identity ztunnel proved at L4 — enforced locally in the waypoint, no external rate-limit server. Only storefront draws from this bucket; checkout presenting the same user JWT is untouched.

kind: EnterpriseAgentgatewayPolicy
spec:
  targetRefs: [{ group: gateway.networking.k8s.io, kind: Gateway, name: petshop-waypoint }]
  traffic:
    rateLimit:
      conditional:
        - condition: 'source.identity.serviceAccount == "storefront"'
          policy:
            local:
              - requests: 5
                unit: Minutes

storefront:    200 200 200 200 200 429 429 429
checkout-blue: 200 200 200 200 200 200 200 200

What Solo Enterprise adds

You've already used every Enterprise piece this page relies on: the Solo distribution mesh, the Gloo UI, the workload claims and the agentgateway waypoint. The differentiators worth naming:

• The Gloo UI — Solo's dashboard (installed earlier): mesh inventory, the workloads and the Observability graph, across a fleet of clusters. The concrete "see it" one.
• L7 telemetry from ztunnel, no waypoint — ztunnel reports HTTP fields (method / path / response_code) through the licensed Solo telemetry pipeline: istioctl ztunnel-config all <ztunnel> -o json | jq .config.l7Config shows metrics, access-log and tracing enabled, pointed at an OTel collector endpoint. The identity-aware L4 access logs above are what ztunnel always writes to stdout.
• Multicluster ambient — east-west gateway (istio-eastwest), istioctl multicluster expose, global-service segments — so identity and JWT policy span clusters.
• agentgateway as the waypoint — you just used it for the whole L7 half: JWT, CEL authorization, canary routing and identity-scoped rate limiting; the same data plane also brings MCP/LLM/AI policy to mesh traffic.
• Lifecycle & packaging — FIPS builds, air-gap, Solo UI; and the Gloo Operator (ServiceMeshController) for fleet lifecycle if you want it (this lab installs with Helm for full control of the values).
• Per-workload cert claims + CEL at L4 — the workload-claims step above.

L4 or L7? the decision

The rows above are the point of the whole lab. Everything in the top half — workload identity, source namespace, ports, CEL conditions, DENY precedence, and per-pod certificate claims — is decided by ztunnel, which is already running on every node: no extra hop, no extra proxy to size, patch and pay for. Workload claims move a whole class of decisions down into that half — telling apart pods that share a ServiceAccount used to force you to L7. A waypoint is an extra proxy in the request path, and in ambient it is opt-in per namespace or per service — so deploy one only where the decision is genuinely about HTTP: a user's token, a method, a canary split, a rate limit. In this lab that was one namespace; everything else stayed at L4.

See also

• Related — Open ports vs used ports on ambient (L4 authz + ztunnel JSON logs)
• Related — Sidecar to Ambient Upgrade
• Solo docs — Solo Enterprise for Istio (install with Helm)
• Istio reference — AuthorizationPolicy

Versions

Built and verified on: