The story: the estate runs Istio in sidecar mode and wants ambient, for the density win of one ztunnel per node instead of a proxy in every pod. And the whole migration is upstream: community images, community Helm charts, community labels. This lab does it end to end on OSS Istio, on one kind cluster, with a load generator running across every cut so you can see the zero in zero downtime. It also shows you, live, the one thing you must plan around on community images: while the fleet is mixed, a caller that still has a sidecar does not route through the destination's waypoint, so the waypoint's L7 policy does not yet apply to that caller.
Two design choices give this lab its shape. First, the switch is a label: after one helm upgrade the mesh speaks both dataplanes, and from then on each namespace chooses sidecar or ambient with a single label, in either direction. Second, one waypoint for the whole cluster: instead of a waypoint per namespace, a single Gateway in a platform namespace serves every L7 namespace that attaches to it, so the L7 fleet you operate is one deployment, not one per team.
The model to hold: in ambient there is no sidecar. ztunnel handles L4 for every enrolled pod (mTLS, L4 authorization, TCP telemetry). Everything the sidecar did at L7 moves to a waypoint. So a namespace with only L4 concerns needs no waypoint; a namespace with any L7 policy needs one before it is enrolled. That single decision drives the whole migration.
The end state
Where the migration lands. Every namespace is on ambient. The two L7 namespaces, petstore and
petstore-orders, sit behind the same waypoint in mesh-infra, which does the
HTTPRoute canary and enforces both GET-only authorization policies. The L4 namespace has only ztunnel. The
calling estate is ambient too, which is what makes the waypoint's policy apply to its calls.
The switch is a label
There are two switches in this migration, and both are small on purpose.
The mesh-level switch makes the mesh bilingual. The cluster starts with plain
base + istiod in sidecar mode. Three helm commands later (istiod upgraded to
profile=ambient, plus the cni and ztunnel charts) the mesh can serve
both dataplanes at once. Nothing migrates at this point: every app pod keeps its sidecar, the load generator
keeps scoring 100%, and ztunnel sits idle because nothing is enrolled.
The namespace-level switch is one label swap. A namespace on sidecars carries
istio-injection=enabled; to migrate it you swap that for
istio.io/dataplane-mode=ambient and roll the deployments so the pods come back without sidecars:
bashthe whole per-namespace migration
kubectl label namespace petstore-data istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n petstore-data rollout restart deploy
And the rollback is the same command run the other way: put istio-injection=enabled back, remove
istio.io/dataplane-mode, roll the deployments, and the namespace is a sidecar namespace again.
The lab does exactly that to petstore-orders near the end, under load, and then flips it forward
a second time. Both directions are routine, which is the property you want before you migrate anything that
matters.
One waypoint for the whole cluster
The default waypoint pattern is one per namespace: istioctl waypoint apply -n petstore creates a
Gateway in petstore and that namespace uses it. Fine at small scale, but it recreates a shape you
just left behind: an Envoy deployment per team, each with replicas, requests and limits, upgrades and
monitoring. If the reason you wanted ambient was fewer proxies to run, a proxy per namespace is not the end
state you had in mind.
Waypoints do not have to be per-namespace. A waypoint is just a Gateway, and a Gateway can admit routes from
anywhere you allow. This lab deploys one waypoint in a platform namespace
(mesh-infra) with allowedRoutes: namespaces.from: All, and every L7 namespace
attaches to it with two labels:
bashattach a namespace to the shared waypoint
kubectl label namespace petstore \
istio.io/use-waypoint=cluster-waypoint \
istio.io/use-waypoint-namespace=mesh-infra
istio.io/use-waypoint names the waypoint; istio.io/use-waypoint-namespace says where
it lives (without it, Istio looks for the waypoint in the workload's own namespace). Policy and routing stay
exactly where they belong. The AuthorizationPolicy and HTTPRoute live in the
workload's namespace and target its Service, so teams keep ownership of their rules while the
platform team owns the one proxy that enforces them. In this lab, petstore and
petstore-orders both attach to it, and you can watch a single Envoy pod enforce two namespaces'
policies.
L4 or L7: the one decision per namespace
Audit each namespace's policies and ask one thing: does anything here operate on the content of a
request? ztunnel is an L4 proxy, and it does a lot with no waypoint at all: mutual TLS, authorization on
identity, namespace and port, TCP telemetry, and L4 load balancing. A namespace whose only concerns are those
migrates with ztunnel alone. In this lab that is petstore-data, whose STRICT mTLS and
identity-based AuthorizationPolicy carry over without editing a single line of YAML.
You only need the waypoint when something looks inside HTTP: an AuthorizationPolicy on methods, paths or
headers, VirtualService or HTTPRoute behaviour (routing, retries, timeouts), JWT rules, per-request telemetry.
Deploy it (or attach to it) before you enrol the namespace, because if you enrol a namespace that has
an L7 selector policy and no waypoint, ztunnel fails safe and denies all traffic to that workload. And the L7
policy needs its shape changed as it moves: a sidecar policy selects pods
(selector.matchLabels), a waypoint policy targets the Service
(targetRefs). Same rule, different attachment:
| Sidecar (before) | Waypoint (after) | |
|---|---|---|
| L7 authz | selector.matchLabels: {app: catalog} — enforced in catalog's own sidecar |
targetRefs: [{kind: Service, name: catalog}] — enforced on the shared waypoint |
| Canary | DestinationRule subsets + VirtualService weights, applied by the caller's sidecar | per-version Services + HTTPRoute weights, enforced on the waypoint (waypoints do not do subset routing) |
| L4 authz / mTLS | selector-based AuthorizationPolicy, PeerAuthentication | unchanged — ztunnel enforces the same objects |
Who enforces your L7 policy today?
This is the question that decides your migration order, and the lab makes you watch the answer change. In
sidecar mode the GET-only rule on catalog is enforced by catalog's own sidecar: any
caller, from anywhere, gets a 403 on a DELETE. Once petstore migrates behind the
waypoint, the rule is enforced by the waypoint. And on community Istio, traffic only routes through
a waypoint when the caller is on the ambient dataplane. A caller still running a sidecar
sends straight to the pods, so for that caller the DELETE that used to be denied comes back 200.
The lab demonstrates this live in STEP 10 and then closes it in STEP 11 by migrating the calling namespace.
The practical rule for OSS ambient: migrate a service's callers in the same window as the
service whenever the service carries L7 security policy, and treat the mixed period as a period where
L7 rules are advisory for unmigrated callers. If you need the mixed period itself to be safe (sidecar and
ingress callers routed through the waypoint so its policy holds for everyone during a long,
team-by-team migration) that interop is what the Solo distribution adds on top of upstream
(ENABLE_WAYPOINT_INTEROP), and the Enterprise edition
of this lab runs the same migration with that behaviour on.
Notes from the field
-
Your sidecars are initContainers now. On a current cluster (Kubernetes 1.29+, this lab runs
1.35) Istio injects
istio-proxyas a native sidecar container: it lives inspec.initContainerswithrestartPolicy: Always, andkubectl get pod -o jsonpath='{.spec.containers[*].name}'will not show it. Scripts that audit "does this pod have a sidecar" must check both lists. The READY column still counts it (2/2). -
Roll every sidecar namespace once, right after ambient arrives. Sidecars injected
before istiod ran with
profile=ambientlackISTIO_META_ENABLE_HBONE=true, so ztunnel can only reach them in plaintext, which STRICT mTLS rejects. One rolling restart per namespace, before any neighbour enrols, re-injects them HBONE-capable. Skip this and the first namespace you migrate loses its sidecar callers. -
ztunnel fails safe on L7 selector policies. Enrol a namespace that still has a
selector-based L7 AuthorizationPolicy and ztunnel denies all traffic to those pods rather than let an HTTP
rule go unenforced. Convert the policy to
targetRefs(waypoint-enforced) before the label flip, then delete the selector version. - Waypoints do not do DestinationRule subset routing. Move any subset-based VirtualService split to per-version Services + an HTTPRoute before its callers go ambient. Do it while everything is still on sidecars: sidecars apply Service-attached HTTPRoutes too, so the cutover is a no-op you can verify safely.
-
Audit for
mode: DISABLEbefore enrolling. STRICT and PERMISSIVE PeerAuthentication carry over to ambient; DISABLE has no ambient equivalent because HBONE is always mTLS. Find any DISABLE carve-outs and retire them first. -
An L4 deny does not reset the client's connection. With a client-side proxy in the path the
client's
connect()succeeds locally, so probing a deny with exit codes lies to you. Probe with a payload instead: sendPINGto redis and check whether+PONGcomes back. Allowed identity: PONG. Denied identity: silence. - Rollback order matters. Going back to sidecars: flip the labels, roll the deployments, and only then restore the selector policy and retire the targetRefs one. Apply the selector policy while the pods are still ambient and ztunnel fail-safes into denying the workload (note 3, in reverse).
-
Zero downtime still needs graceful shutdown. Every migration step here is a rolling
restart, and a server that dies instantly on SIGTERM will reset whatever keep-alive connections it is
holding, mesh or no mesh. The app deployments carry a
preStop: sleep: {seconds: 5}(the native sleep action, so it works in scratch images with no shell) so the endpoint deregisters and drains before the process stops. Without it, expect an occasional blip of a request or two per cut; the mesh gets blamed, but the pod lifecycle is the culprit.
Run the lab, step by step
This is what you run, top to bottom, from a clone of the lab repo (in the
istio-ambient-migration-oss-kind directory). Each green STEP has the change to
run (open view the YAML to see exactly what goes in) and the kubectl or
curl to prove it, with expected output in comments. Nothing needs a licence, a registry login or
a cloud account; the only tools are Docker, kind, kubectl, helm and
istioctl. A fortio load generator runs in petstore-clients; a bounded fortio run at
each cut should read 100% 200 once the mesh reconverges (allow a few seconds after a restart or
a route change before you measure). If you would rather watch it than type it,
./scripts/e2e.sh runs this whole sequence with assertions.
Create the kind cluster and install the Gateway API CRDs
bashrun
kind create cluster --config kind/cluster.yaml
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yamlbashverify
kubectl get nodes
# ambient-oss-control-plane Ready control-plane (3 nodes total, all Ready)
kubectl get crd gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io
# both CRDs listedyamlview the YAML · kind/cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ambient-oss
nodes:
- role: control-plane
- role: worker
- role: workera 3-node kind cluster (1 control-plane, 2 workers) and the Gateway API CRDs installed.
Load the upstream Istio images into kind
Straight from Docker Hub, no registry auth. Pre-loading keeps the later helm installs fast and deterministic.
bashrun
ARCH=$(uname -m | sed 's/aarch64/arm64/;s/x86_64/amd64/')
for img in pilot proxyv2 install-cni ztunnel; do
ref="docker.io/istio/$img:1.30.3"
docker pull -q "$ref"
docker save --platform "linux/$ARCH" "$ref" -o "/tmp/$img.tar"
kind load image-archive "/tmp/$img.tar" --name ambient-oss
donebashverify
docker exec ambient-oss-control-plane crictl images | grep -c docker.io/istio
# 4 (pilot, proxyv2, install-cni, ztunnel on the node)the four upstream Istio images present on every kind node.
Install Istio in plain sidecar mode
This is the estate you are migrating from: base (CRDs) + istiod, no ambient components at all.
bashrun
helm upgrade --install istio-base base \
--repo https://istio-release.storage.googleapis.com/charts \
-n istio-system --create-namespace --version 1.30.3 \
--set defaultRevision=default --wait
helm upgrade --install istiod istiod \
--repo https://istio-release.storage.googleapis.com/charts \
-n istio-system --version 1.30.3 --wait \
--set global.hub=docker.io/istio --set global.tag=1.30.3 \
--set meshConfig.accessLogFile=/dev/stdoutbashverify
kubectl -n istio-system get deploy istiod
# istiod 1/1
kubectl -n istio-system get ds
# No resources found (no CNI node agent, no ztunnel — sidecar mode only)istiod Running, and no DaemonSets: the ambient dataplane does not exist yet.
Deploy the petstore app and its sidecar-mode policies
Four namespaces, all labelled istio-injection=enabled: petstore (catalog v1/v2 +
DR/VS canary + GET-only authz), petstore-orders (a second L7 service with its own GET-only
rule), petstore-data (redis + identity-based L4 authz), petstore-clients
(checkout + fortio). Plus mesh-wide STRICT mTLS.
bashrun
kubectl apply -f yaml/00-namespaces.yaml
kubectl apply -f yaml/10-apps/
kubectl apply -f yaml/20-policies-sidecar/
kubectl -n petstore rollout status deploy/catalog-v1 deploy/catalog-v2 deploy/data-client --timeout=300s
kubectl -n petstore-orders rollout status deploy/orders --timeout=180s
kubectl -n petstore-data rollout status deploy/redis --timeout=180s
kubectl -n petstore-clients rollout status deploy/checkout deploy/fortio --timeout=180sbashverify — the sidecar estate works and enforces
# native sidecars: istio-proxy is an INIT container on k8s 1.29+ (READY still says 2/2)
kubectl -n petstore get pod -l app=catalog -o jsonpath='{.items[0].spec.initContainers[*].name}'; echo
# istio-init istio-proxy
# the GET-only rule, enforced by catalog's own sidecar
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' http://catalog.petstore/
# 200
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' -X DELETE http://catalog.petstore/
# 403
# the canary is 100% v1 (the caller's sidecar applies the VirtualService)
kubectl -n petstore-clients exec deploy/checkout -c checkout -- sh -c \
'i=0; while [ $i -lt 30 ]; do curl -s http://catalog.petstore/; i=$((i+1)); done' | grep -c '"version":"v2"'
# 0
# redis: the allowed identity gets PONG, the checkout identity gets silence (L4 deny)
kubectl -n petstore logs deploy/data-client --tail=1
# PONG
kubectl -n petstore-clients exec deploy/checkout -c checkout -- sh -c 'printf "PING\r\n" | nc -w 3 redis.petstore-data 6379'
# (no output — denied)
# the zero-downtime scoreboard
kubectl -n petstore-clients exec deploy/fortio -c fortio -- fortio load -c 4 -qps 25 -t 10s -quiet http://catalog.petstore/ | grep 'Code '
# Code 200 : 250 (100.0 %)a fully working sidecar estate: canary pinned to v1, DELETE denied 403, redis answering only the catalog identity, fortio at 100%.
The mesh-level switch: the ambient dataplane arrives, under load
Start a fortio run first, then make the change while it flies: istiod to profile=ambient, plus
the cni and ztunnel charts. Sidecars keep serving throughout. Nothing is enrolled
yet.
bashrun
# 150s of load in the background — it flies across the whole standup
kubectl -n petstore-clients exec deploy/fortio -c fortio -- \
fortio load -c 4 -qps 25 -t 150s -quiet http://catalog.petstore/ > /tmp/fortio-ambient.txt &
helm upgrade istiod istiod \
--repo https://istio-release.storage.googleapis.com/charts \
-n istio-system --version 1.30.3 --wait --reuse-values \
--set profile=ambient --set istio_cni.enabled=true
helm upgrade --install istio-cni cni \
--repo https://istio-release.storage.googleapis.com/charts \
-n istio-system --version 1.30.3 --wait \
--set profile=ambient --set global.hub=docker.io/istio --set global.tag=1.30.3 \
--set 'excludeNamespaces[0]=istio-system' --set 'excludeNamespaces[1]=kube-system'
helm upgrade --install ztunnel ztunnel \
--repo https://istio-release.storage.googleapis.com/charts \
-n istio-system --version 1.30.3 --wait \
--set profile=ambient --set hub=docker.io/istio --set tag=1.30.3
wait # for the fortio run to finishbashverify
kubectl -n istio-system get ds
# istio-cni-node 3/3
# ztunnel 3/3 (one per node)
grep 'Code ' /tmp/fortio-ambient.txt
# Code 200 : 3750 (100.0 %) — zero dropped requests across the standupthe mesh is bilingual: sidecars still everywhere and still serving, ztunnel on every node with nothing enrolled, fortio at 100%.
Roll every app namespace once (HBONE re-injection)
Sidecars injected before the ambient profile lack ISTIO_META_ENABLE_HBONE, so ztunnel could
only reach them in plaintext, which STRICT mTLS rejects. One roll, before anything enrols, and every
re-injected sidecar can accept HBONE from ambient callers.
bashrun
for ns in petstore petstore-orders petstore-data petstore-clients; do
kubectl -n $ns rollout restart deploy
kubectl -n $ns get deploy -o name | xargs -I{} kubectl -n $ns rollout status {} --timeout=300s
donebashverify
kubectl -n petstore get pod -l app=catalog -o jsonpath='{.items[0].spec.initContainers[?(@.name=="istio-proxy")].env[?(@.name=="ISTIO_META_ENABLE_HBONE")].value}'; echo
# trueevery sidecar advertises HBONE. This is the step people skip and then blame ambient for.
Migrate the L4-only namespace: one label, no waypoint
petstore-data has STRICT mTLS and an identity-based L4 policy, nothing L7. ztunnel enforces
both, so the namespace migrates with the label flip alone and zero YAML changes.
bashrun
kubectl label namespace petstore-data istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n petstore-data rollout restart deploy/redis
kubectl -n petstore-data rollout status deploy/redis --timeout=180sbashverify
# no sidecar left (no istio-proxy init container either)
kubectl -n petstore-data get pod -l app=redis -o jsonpath='{.items[0].spec.containers[*].name} {.items[0].spec.initContainers[*].name}'; echo
# redis
# the allowed identity still gets PONG — same policy, now enforced by ztunnel
kubectl -n petstore logs deploy/data-client --tail=1
# PONG
# the denied identity still gets silence
kubectl -n petstore-clients exec deploy/checkout -c checkout -- sh -c 'printf "PING\r\n" | nc -w 3 redis.petstore-data 6379'
# (no output — denied)
istioctl ztunnel-config workload | grep redis
# petstore-data redis-... ... ambient-oss-worker... None HBONEredis on ambient with the identical selector-based L4 AuthorizationPolicy still enforcing: the PONG stream never blinks.
Modernise the canary: DR/VS subsets to HTTPRoute
Waypoints do not do subset routing, so the version split must become per-version Services + an HTTPRoute before catalog's callers go ambient. Do it now, while everything relevant is still on sidecars: sidecars apply Service-attached HTTPRoutes too, so this cutover is a traffic no-op you can verify.
bashrun
kubectl apply -f yaml/50-httproute/10-versioned-services.yaml
kubectl apply -f yaml/50-httproute/20-catalog-httproute.yaml
kubectl apply -f yaml/50-httproute/30-catalog-dr-nosubset.yaml
kubectl -n petstore delete virtualservice catalogbashverify
# still 100% v1 — the HTTPRoute starts with the same weights the VS had
kubectl -n petstore-clients exec deploy/checkout -c checkout -- sh -c \
'i=0; while [ $i -lt 30 ]; do curl -s http://catalog.petstore/; i=$((i+1)); done' | grep -c '"version":"v2"'
# 0
kubectl -n petstore-clients exec deploy/fortio -c fortio -- fortio load -c 4 -qps 25 -t 10s -quiet http://catalog.petstore/ | grep 'Code '
# Code 200 : 250 (100.0 %)yamlview the YAML · yaml/50-httproute/20-catalog-httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: catalog
namespace: petstore
spec:
parentRefs:
- group: ""
kind: Service
name: catalog
port: 80
rules:
- backendRefs:
- name: catalog-v1
port: 80
weight: 100
- name: catalog-v2
port: 80
weight: 0the split now lives in a Gateway API HTTPRoute attached to the catalog Service; traffic unchanged.
Deploy THE waypoint: one for the whole cluster
bashrun
kubectl apply -f yaml/30-waypoint/cluster-waypoint.yaml
kubectl -n mesh-infra wait --for=condition=Programmed gateway/cluster-waypoint --timeout=120s
kubectl -n mesh-infra rollout status deploy/cluster-waypoint --timeout=180sbashverify
kubectl -n mesh-infra get gateway,pods
# gateway.../cluster-waypoint istio-waypoint 10.96.x.x True (Programmed)
# pod/cluster-waypoint-... 1/1 Running ← ONE pod, the whole cluster's L7yamlview the YAML · yaml/30-waypoint/cluster-waypoint.yaml
apiVersion: v1
kind: Namespace
metadata:
name: mesh-infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: cluster-waypoint
namespace: mesh-infra
labels:
istio.io/waypoint-for: service
spec:
gatewayClassName: istio-waypoint
listeners:
- name: mesh
port: 15008
protocol: HBONE
allowedRoutes:
namespaces:
from: All # ← the line that makes it cluster-widea Programmed istio-waypoint Gateway and a single Envoy pod in mesh-infra. Nothing routes through it yet.
Migrate both L7 namespaces behind the shared waypoint, under load
Policy shape first (targetRefs in, selector out; ztunnel fail-safes if you skip this), then the label
flips: use-waypoint + use-waypoint-namespace + dataplane-mode=ambient
on both namespaces, and a roll.
bashrun
kubectl apply -f yaml/40-policies-waypoint/
kubectl -n petstore delete authorizationpolicy catalog-get-only
kubectl -n petstore-orders delete authorizationpolicy orders-get-only
kubectl -n petstore-clients exec deploy/fortio -c fortio -- \
fortio load -c 4 -qps 25 -t 120s -quiet http://catalog.petstore/ > /tmp/fortio-waypoint.txt &
for ns in petstore petstore-orders; do
kubectl label namespace $ns \
istio.io/use-waypoint=cluster-waypoint \
istio.io/use-waypoint-namespace=mesh-infra \
istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n $ns rollout restart deploy
done
kubectl -n petstore get deploy -o name | xargs -I{} kubectl -n petstore rollout status {} --timeout=300s
kubectl -n petstore-orders rollout status deploy/orders --timeout=180s
waitbashverify — one proxy, two namespaces
grep 'Code ' /tmp/fortio-waypoint.txt
# Code 200 : 3000 (100.0 %) — zero dropped requests across the enrolment
# ztunnel routes both namespaces' services via the ONE waypoint
istioctl ztunnel-config service | grep -E 'catalog|orders'
# petstore catalog 10.96.x.x cluster-waypoint 4/4
# petstore catalog-v1 10.96.x.x cluster-waypoint 2/2
# petstore catalog-v2 10.96.x.x cluster-waypoint 2/2
# petstore-orders orders 10.96.x.x cluster-waypoint 2/2
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' http://catalog.petstore/
# 200
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' http://orders.petstore-orders/
# 200bashverify — the community mixed-fleet gap, live
# checkout still has a sidecar, and community sidecars do NOT route through waypoints.
# The GET-only rule now lives on the waypoint — so for THIS caller it no longer applies:
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' -X DELETE http://catalog.petstore/
# 200 ← was 403 in STEP 4. The policy did not break; this caller bypasses its enforcement point.both L7 namespaces served by the single waypoint at 100%, and the honest OSS caveat on display: an unmigrated caller is outside the waypoint's law until its own namespace flips (next step). On the Solo distribution, sidecar and ingress callers are routed through the waypoint instead, which is what the Enterprise edition shows.
Migrate the calling estate: the waypoint's policy now applies to it
bashrun
kubectl label namespace petstore-clients istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n petstore-clients rollout restart deploy
kubectl -n petstore-clients rollout status deploy/checkout deploy/fortio --timeout=180sbashverify
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' -X DELETE http://catalog.petstore/
# 403 ← the gap is closed: ambient callers route through the waypoint
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' -X DELETE http://orders.petstore-orders/
# 403 ← same single waypoint enforcing the second namespace's rule
kubectl -n petstore-clients exec deploy/fortio -c fortio -- fortio load -c 4 -qps 25 -t 10s -quiet http://catalog.petstore/ | grep 'Code '
# Code 200 : 250 (100.0 %)every namespace ambient, every L7 rule enforced again, by one proxy.
Shift the canary live at the waypoint
The HTTPRoute weights are now enforced by the waypoint. Shift them with a patch, watch v2 take half the traffic, then pin back.
bashrun
kubectl -n petstore patch httproute catalog --type=json -p='[
{"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":50},
{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":50}
]'bashverify
kubectl -n petstore-clients exec deploy/checkout -c checkout -- sh -c \
'i=0; while [ $i -lt 30 ]; do curl -s http://catalog.petstore/; i=$((i+1)); done' | grep -c '"version":"v2"'
# ~15 (half of 30 — the live 50/50 split)
# pin back to v1 before the rollback step
kubectl -n petstore patch httproute catalog --type=json -p='[
{"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":100},
{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}
]'a zero-downtime weight shift, done by the same knob Argo Rollouts would drive.
Roll back a namespace, and forward again
The switch has to work in both directions before you trust it. petstore-orders goes back to
sidecars: labels flipped back, one roll, selector policy restored after the pods are sidecars
again (never before; field note 7). Then forward a second time.
bashrun — back to sidecars
kubectl label namespace petstore-orders istio-injection=enabled \
istio.io/dataplane-mode- istio.io/use-waypoint- istio.io/use-waypoint-namespace- --overwrite
kubectl -n petstore-orders rollout restart deploy/orders
kubectl -n petstore-orders rollout status deploy/orders --timeout=180s
# pods are sidecars again → restore the selector policy, retire the targetRefs one
kubectl apply -f yaml/20-policies-sidecar/40-l7-authz-orders.yaml
kubectl -n petstore-orders delete authorizationpolicy orders-get-only-waypointbashverify
kubectl -n petstore-orders get pod -l app=orders -o jsonpath='{.items[0].spec.initContainers[*].name}'; echo
# istio-init istio-proxy ← the sidecar is back
kubectl -n petstore-clients exec deploy/checkout -c checkout -- curl -s -o /dev/null -w '%{http_code}\n' -X DELETE http://orders.petstore-orders/
# 403 ← enforced by the sidecar againbashrun — and forward again
kubectl apply -f yaml/40-policies-waypoint/20-orders-authz-targetref.yaml
kubectl -n petstore-orders delete authorizationpolicy orders-get-only
kubectl label namespace petstore-orders \
istio.io/use-waypoint=cluster-waypoint \
istio.io/use-waypoint-namespace=mesh-infra \
istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n petstore-orders rollout restart deploy/orders
kubectl -n petstore-orders rollout status deploy/orders --timeout=180srollback and re-enrolment are the same routine label flip. fortio against catalog reads 100% throughout: the other namespaces never notice.
The end state, then teardown
bashverify
# zero sidecars anywhere in the app namespaces
for ns in petstore petstore-orders petstore-data petstore-clients; do
kubectl -n $ns get pods -o jsonpath='{range .items[*]}{.spec.initContainers[*].name}{" "}{end}'
done | tr ' ' '\n' | grep -c istio-proxy
# 0
# exactly ONE waypoint pod for the whole cluster; ztunnel on every node
kubectl get pods -A -l gateway.networking.k8s.io/gateway-name
# mesh-infra cluster-waypoint-... 1/1 Running (just the one)
kubectl -n istio-system get ds ztunnel
# ztunnel 3/3bashteardown
kind delete cluster --name ambient-ossthe diagram at the top of this page, live: one waypoint, three ztunnels, no sidecars.
Migration checklist
The lab, compressed into the checks you would run against a real cluster.
- Before anything migrates
- Classify every namespace L4 or L7. Anything that reads HTTP content (methods, paths, headers, VS routing, JWT) is L7 and needs a waypoint attachment before enrolment; identity/port-only policy is L4 and needs nothing.
- Audit PeerAuthentication for
mode: DISABLE: no ambient equivalent; retire the carve-outs first. - If the mesh CA is external (cert-manager istio-csr, Vault), check it will sign ztunnel. ztunnel only generates ECDSA P-256 keys, so a signing role locked to RSA rejects every ambient certificate, and istio-csr needs
app.server.caTrustedNodeAccountsfor the ambient trust model. Sidecars to Ambient When Your PKI Is RSA rehearses exactly that failure and the fix. - Find DR subset routing and plan its move to per-version Services + HTTPRoute; do the cutover while still on sidecars, where it is a no-op.
- Decide the waypoint topology: shared cluster-wide, per-namespace, or a mix. Shared needs
allowedRoutes: namespaces.from: All(or a Selector) on the Gateway. - The switch
- Enable ambient with helm (istiod
profile=ambient, cni, ztunnel) under load; sidecars are untouched. - Roll every sidecar namespace once so re-injected sidecars advertise HBONE, before any neighbour enrols.
- Per namespace, in order
- L4 namespaces first: label flip + roll; verify the same policies enforce (allowed identity still served, denied identity still silent).
- L7 namespaces: convert selector policies to
targetRefs, attach the namespace to its waypoint, then flip the label and roll. - Migrate callers in the same window as L7-protected services: on community images a sidecar caller bypasses the waypoint, so its policy view lapses until its namespace flips.
- Keep a load generator running and read it after every change. It should say 100%; if it does not, stop and find out why before you touch the next namespace.
- Safety net
- Rehearse the rollback: labels back, roll, selector policy restored after the pods are sidecars again. If you have not rehearsed it, you do not have it.
How Enterprise Istio helps with this migration
Everything above runs on community Istio, and for a small estate that can migrate all of a service's callers in one window, it is enough. The places a real fleet feels the difference are the mixed period, the audit, the lifecycle and the visibility while you cut. This is what Solo Enterprise for Istio adds to each stage of the exact migration you just ran:
-
The mixed period stays enforced (the STEP 10 gap). On the Solo images, a sidecar or the
Istio ingress gateway that sees its destination uses a waypoint forwards to the waypoint over HBONE, so the
waypoint's L7 policy runs for callers that have not migrated yet (
ENABLE_WAYPOINT_INTEROP, on by default). The DELETE that came back200in STEP 10 comes back403on Enterprise, which means you can migrate namespaces team by team, over months, without a window where L7 rules are advisory. This is the single biggest difference for a long migration, and the Enterprise edition of this lab demonstrates it live. -
The audit is generated, not hand-rolled. The checklist above starts with "classify every
namespace L4 or L7". The
gloo ambient estimateandgloo ambient migrateCLI reads the cluster and does that classification for you: which namespaces need a waypoint, which policies need the selector to targetRefs transform, what carries over untouched. -
The switch is one field, managed. STEPs 3 and 5 sequence three helm releases by hand and
you own their upgrades forever. The Gloo Operator runs the same mesh from a single
ServiceMeshControllerresource, and the sidecar-to-ambient switch isdataplaneMode: Sidecar→Ambient. Upgrades, FIPS (BoringCrypto) builds and the revision mechanics ride the same declarative resource. -
You can see both dataplanes while you cut. The Gloo UI service graph shows sidecar and
ambient workloads in one picture during the mixed period. And the Solo ztunnel emits HTTP-level telemetry
with no waypoint at all, so an L4-only namespace like
petstore-datakeeps request-level metrics and access logs after migrating, where this lab's visibility drops to TCP. - Sharper identity at L4. Upstream ztunnel authorises on the ServiceAccount identity, so two workloads sharing a ServiceAccount are indistinguishable. The Solo images add workload claims: pod-level attributes carried in the certificate that ztunnel can authorise on with no waypoint. Certificates as workload identity in ambient runs that end to end.
- Support that matches how long migrations actually take. Community minors drop out of support quickly; Solo maintains an N-4 patched window with CVE backports, so the version you started the migration on is still patched when the last team finishes.
- The phase after this one. Once the cluster is ambient, the next ask is usually more clusters: east-west HBONE peering, global services and locality failover. Multi-region load balancing and failover on ambient is that phase, on the Solo distribution.
See also
- Sidecar to Ambient Upgrade: the Enterprise edition of this migration, installed by the Gloo Operator, with the mixed-fleet interop (sidecar and ingress traffic routed through the waypoint) that keeps L7 policy enforced during the mixed period.
- Vault + istio-csr: sidecars on RSA, ambient on EC: what happens to this same migration when the mesh CA is external and the PKI is RSA-only.
- Certificates as workload identity in ambient: the L4 identity model this lab's Redis policy relies on, in depth.
Versions
Built and verified on:
v1.5.11.30.31.35