Anthos Service Mesh to Enterprise Istio Ambient, on GKE

Why this is a migration and not an upgrade

Two things make this urgent, and both come from Google's own published positions.

The version underneath is already off the supported list. The clusters in this lab report 1.20.8-asm.88. Google's version page now lists 1.27, 1.28 and 1.29 as the supported in-cluster lines, most recently 1.29.5-asm.12 released on 15 July 2026, and puts the whole 1.20 line under unsupported in-cluster versions. Anything on 1.20 is several lines behind the floor, and Google sets the pace at which that gap widens.

The provisioning path many estates were onboarded through is gone. Provisioning managed Cloud Service Mesh with asmcli on Google Cloud clusters was deprecated on 22 August 2024 and support for it ended in February 2025. Existing deployments were not switched off, and clusters outside Google Cloud carry on with asmcli as before, but the onboarding route is closed and the supported paths are gcloud and the console.

Add the thing this lab is actually about. ASM does not offer you ambient, the control plane that decides your Istio version belongs to Google, and its API server already serves older CRD versions than a current Istio expects, which you can see in the table further down. There is no in-place step from here to ambient. So the move is sideways: a new cluster, a mesh you own, and a drain.

Sources: Cloud Service Mesh versions, Provision managed Cloud Service Mesh with asmcli. Both read on 10 August 2026.

Two moves, not one

This is the thing to get straight before any of the commands make sense, because a migration off ASM is two separate changes and people run them together and then cannot tell which one broke.

They are independent, and you can run them in either order. This lab does cluster first, dataplane second: the workloads are deployed onto the new cluster in sidecar mode, proven there against the same tests they pass on ASM, and only then migrated to ambient. That order is deliberate. If you flip the dataplane on a cluster where the app has never run, a failure has two possible causes and you get to guess which. Doing it this way, each step has one variable.

It also means the two clusters have very different workloads. Steps 1 to 11 below all happen on the new cluster. The ASM cluster is touched in exactly one step, and only if you need the bridge.

The environment

Two clusters already run ASM and are never modified at the control-plane level: no version change, no mesh-wide policy change, no fleet reconfiguration. Everything this lab builds lands on new clusters. That boundary keeps the exercise repeatable, and it is the boundary you want on a real migration too: experiments stay off the clusters carrying production traffic.

There is a fourth cluster in this lab that is not in the diagram, because it is not part of the architecture: a plain community Istio cluster, identical in every other respect, used purely as a control. The migration runs twice, once on each, and the two results are compared at the end. That comparison is what settles the mixed-fleet interop question.

Do not register the new cluster into the fleet

Check the fleet's default member config before you create anything:

gcloud container fleet mesh describe --project "$PROJECT"

If fleetDefaultMemberConfig.mesh.management reads MANAGEMENT_AUTOMATIC, then registering the new cluster into that fleet makes Google start managing a mesh on it too. You would install Solo Istio onto a cluster that Google is simultaneously handing an ASM install, which is the exact CRD collision the whole migration exists to escape, self-inflicted on the new cluster instead of inherited on the old one. Simplest answer: do not register the target cluster into the fleet at all. Fleet registration is opt-in later if you specifically want fleet-scoped observability.

What is configured where

Start with what you can skip. If you can move a service and everything that calls it in the same window, you need nothing on the ASM side at all. Move the namespace, repoint its callers, done. The east-west bridge below exists for one situation: a service has moved and some of its callers have not, so a call has to cross while both clusters are live. Plenty of estates have that situation and some do not, so treat the whole left-hand column as conditional.

When you do need it, the ASM side gets one namespace holding an east-west gateway and one ServiceEntry per crossing call. No remote secret, no injection change, no mesh-wide policy edit, nothing that touches a running sidecar, and nothing at all at the control-plane level.

Two load balancers, doing different jobs. The east-west pair are internal LoadBalancers, one per cluster, reachable only inside the VPC, carrying migration-window traffic between the two meshes. The north-south ingress on the new cluster is a separate, external LoadBalancer, and it is what you eventually point real client traffic at. Keep them apart. Putting east-west traffic on an external LB is the hairpin problem described above, and putting client traffic on the internal one means nobody outside the VPC can reach you.

Why an Istio gateway on the ASM side and agentgateway on the new one

The two sides get different gateways, and not for consistency's sake. On the new cluster there is no reason to run a second gateway stack: Enterprise agentgateway is already installed as the ingress, so the east-west listener is just another Gateway on the same enterprise-agentgateway class, with the internal-LB annotation set through Gateway API's own spec.infrastructure.annotations and an HTTPRoute per exposed service. One gateway product, one control plane, two listeners.

On the ASM cluster it has to be an Istio gateway, for two reasons. First, that cluster has no Gateway API CRDs at all: kubectl get gatewayclass answers the server doesn't have a resource type "gatewayclass", so a Gateway-API gateway cannot even be expressed there without installing CRDs and a controller. Second, and more to the point, an Istio gateway is programmed by the control plane that is already running. Its logs read connected to upstream XDS server: meshconfig.googleapis.com:443, which is ASM's own managed control plane configuring it. Nothing new gets installed into the cluster you promised not to touch, and that is also precisely why the bridge needs no remote secret: the gateway talks to its own local control plane, exactly as every sidecar on that cluster already does.

What GKE requires

Three of the things that cost time on the first pass are decided by GKE, not by Istio. Each one fails in a way that points at the wrong culprit, so they are worth having in front of you before you start.

1. The Gateway API CRDs are a separate install, and they go first. Waypoints are Gateway API Gateways and the Enterprise agentgateway ingress is a GatewayClass, so both fail without them. GKE does not ship them by default in this configuration. Install gateway-api v1.5.1 standard-install.yaml before the first Helm chart, on every cluster. Make it a fixed first step.
2. global.platform=gke on the cni and ztunnel charts. Without it the CNI install fails with insufficient quota to match these scopes: [{PriorityClass In [system-node-critical …]}], and there is no ResourceQuota object anywhere that explains it, because it is GKE itself restricting those priority classes to its own managed namespaces. Moving the chart into kube-system gets past that and straight into a second failure, open /host/opt/cni/bin/istio-cni.tmp…: read-only file system, because the default CNI binary path does not match Container-Optimized OS. The flag fixes both at once: it enables the chart's bundled GKE platform profile, which creates the quota GKE wants and detects the right CNI path, and it lets the chart stay in istio-system where you wanted it.
3. East-west traffic wants an internal LoadBalancer. A pod calling another cluster's external LoadBalancer IP from inside the same VPC can silently time out: GCP's legacy target-pool Network Load Balancers have a hairpin limitation for exactly that shape of traffic. Internal LBs do not, and for traffic that should never leave the VPC they are the right choice anyway. Set the annotation on the first apply. Annotating an already-provisioning external LB afterwards does not cleanly switch it, because the finalizer stays with the external-LB controller, and you end up deleting and recreating the Service.

Where ASM's older CRD versions bite

The ASM control plane in this environment serves Istio 1.20.8. Ask its API server what it actually serves and the picture is uneven:

This is also why the new cluster's control plane must not be pointed at the ASM cluster's Kubernetes API. A remote secret makes istiod watch the other cluster's registry, and on this skew that watch fails outright: failed to list *v1.WorkloadEntry: the server could not find the requested resource. Worse, while a broken remote secret is installed, istiod's readiness probe fails and stays failed. A single bad remote cluster takes the whole control plane's readiness down, not just discovery for that one cluster. Check what your API server serves before you write a manifest:

for c in virtualservices destinationrules serviceentries peerauthentications authorizationpolicies; do
  printf '%-24s %s\n' "$c" \
    "$(kubectl get crd ${c}.networking.istio.io ${c}.security.istio.io \
        -o jsonpath='{.items[*].spec.versions[*].name}' 2>/dev/null)"
done

The order that matters

Note. A namespace with an L7 AuthorizationPolicy needs a waypoint before it is enrolled, and the old selector-based policy has to be converted to a targetRefs policy that the waypoint enforces. The conversion itself is mechanical:

# before, enforced by the pod's own sidecar
selector:
  matchLabels: { app: catalog }

# after, enforced on the waypoint
targetRefs:
  - kind: Service
    group: ""
    name: catalog

The timing of the delete is the part that catches people. The instinct, and what most runbooks say, is to delete it last so the rule is never absent. Put a continuous load generator across the whole sequence instead of sampling after each step and that instinct turns out to be wrong, expensively:

The reason is that a selector-based policy attaches to the workload, so once the workload is ambient it is ztunnel that has to evaluate it. ztunnel is L4. It cannot evaluate an HTTP method rule, so it fails closed, and the waypoint in front returns 503 URX,UC upstream_reset_before_response_started{connection_termination} for every request until you get to the delete. I confirmed it in isolation instead of inferring it from the migration: with the workload already fully ambient and stable at 100%, re-applying that one selector policy produced 995 errors out of 2248 requests in the 42 seconds it was present, and removing it restored 100% immediately. A plain rolling restart of the same pods with no policy change at all scored 3748 (100.0 %), so the pod roll is not the cause.

The fix is to move one step earlier. The waypoint is already enforcing the converted policy from the moment it is programmed and the namespace carries istio.io/use-waypoint, so deleting the old one before enrolment leaves no gap on the Solo distribution. The waypoint's own access log shows it serving requests ten seconds before the first ambient pod existed, while every catalog pod still had a sidecar, and DELETE returned 403 throughout.

What gets deployed, and what each piece is for

Keeping fortio and checkout in a namespace that never migrates is deliberate. The measurement comes from outside whatever is being changed, and there is always a genuine sidecar caller in the fleet to test policy against. Verify that it really is one, with the full pod JSON, and read both container lists:

kubectl -n petstore-legacy get pod "$POD" -o json | python3 -c "
import json,sys
p=json.load(sys.stdin)
print('containers    :', [c['name'] for c in p['spec']['containers']])
print('initContainers:', [(c['name'], c.get('restartPolicy')) for c in p['spec'].get('initContainers',[])])
for c in p['status'].get('containerStatuses',[]) + p['status'].get('initContainerStatuses',[]):
    print(' ', c['name'], 'ready=', c.get('ready'))
"

On the 1.30.3 line, both the Solo distribution and community Istio inject istio-proxy as a Kubernetes native sidecar: an init container with restartPolicy: Always, whose live status lives in status.initContainerStatuses. A check that only reads spec.containers and status.containerStatuses reports no sidecar at all and sends you chasing an injection bug that is not there. jsonpath range expressions and -o custom-columns both produced that false negative on earlier passes. The full-JSON version above never has.

The mesh CA, if yours is RSA-only

Most estates that have run a mesh for years have an external CA and an RSA policy. Sidecars send RSA CSRs, so nothing has ever had to care. ztunnel only generates ECDSA P-256 keys. The first time a pod is enrolled into ambient, the CA is asked to sign something it has never been asked for, and if the signing role is locked to RSA it says no:

Ready = False | Failed | Vault failed to sign certificate: failed to sign certificate by vault:
  Error making API request.
  URL: POST http://vault.vault.svc:8200/v1/pki_int/sign/istio-ca
  Code: 400. Errors:
  * role requires keys of type rsa

The fix is one vault write to set key_type=any on that role, and it pays to be precise with whoever owns the PKI about its scope. It relaxes a single signing path and leaves the rest of the CA alone. ECDSA P-256 is a strength upgrade over RSA-2048. And the RSA workloads keep getting RSA certificates from the same chain, which you can show by capturing a sidecar's leaf certificate serial before and after.

One more ordering rule in this area, because it cost a run: after pointing an already running istiod at a new CA, restart it immediately. The MutatingWebhookConfiguration's caBundle does not refresh from the Helm upgrade alone, so injection breaks with x509: certificate signed by unknown authority the next time a pod starts. A cold install never hits this, which is exactly why it is missing from most install guides.

Which parts of this apply to you

Not all of it will. Four of the twelve build steps are conditional, and skipping the ones you do not need is the difference between a week and an afternoon.

The unconditional core is smaller than the page length suggests: build the cluster, install the mesh in sidecar mode, deploy and prove the workloads, turn on ambient, then migrate namespace by namespace. On this run the per-namespace work was the fast part. An L4 namespace took 3 seconds of wall clock to cut over, the full waypoint-first L7 sequence took about 40 seconds of actual operations, and a rollback took 17 seconds. The load windows in the steps below are long because they are there to catch failures, not because the work is slow.

Run the migration

Every step carries a badge saying which cluster it runs on. Almost all of them are the new cluster: steps 1 to 11 build it, migrate it and prove north-south, step 12 is the community-Istio control if you want the comparison, step 13 is the only one that applies anything to the ASM cluster, and step 14 is the rollback you should practise before you need it. If your services and their callers can move together, you never run step 13 at all.

Set these once. Everything below assumes them, and every command should carry an explicit --context so a step never lands on the wrong cluster.

export PROJECT=your-project-id
export REGION=us-central1
export TARGET=ambient-target          # the Solo Enterprise cluster
export CONTROL=oss-istio-ab           # the community Istio control cluster
export SOURCE=fleet-cluster-1         # the existing ASM cluster
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

# your Solo Istio and agentgateway licence keys
export SOLO_ISTIO_LICENSE_KEY=...
export AGENTGATEWAY_LICENSE_KEY=...

gcloud container clusters create "$TARGET" \
  --project "$PROJECT" --region "$REGION" --node-locations "${REGION}-a" \
  --cluster-version 1.34.9-gke.1065000 \
  --machine-type e2-standard-4 --num-nodes 3 \
  --network default --subnetwork default

gcloud container clusters get-credentials "$TARGET" --region "$REGION" --project "$PROJECT"

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

kubectl get crd gateways.gateway.networking.k8s.io \
  -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'

helm install istio-base oci://us-docker.pkg.dev/soloio-img/istio-helm/base \
  --version 1.30.3-solo -n istio-system --create-namespace --wait

helm install istiod oci://us-docker.pkg.dev/soloio-img/istio-helm/istiod \
  --version 1.30.3-solo -n istio-system \
  --set global.hub=us-docker.pkg.dev/soloio-img/istio \
  --set global.tag=1.30.3-solo \
  --set license.value="$SOLO_ISTIO_LICENSE_KEY" \
  --set meshConfig.accessLogFile=/dev/stdout \
  --wait

kubectl -n istio-system get pods

helm install agentgateway-crds \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
  --version v2026.5.1 -n agentgateway-system --create-namespace --wait

helm install agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --version v2026.5.1 -n agentgateway-system \
  --set licensing.licenseKey="$AGENTGATEWAY_LICENSE_KEY" \
  --set clusterName="$TARGET" \
  --wait

kubectl get gatewayclass

helm install cert-manager jetstack/cert-manager -n cert-manager \
  --create-namespace --version v1.21.1 --set crds.enabled=true --wait

helm install vault hashicorp/vault -n vault --create-namespace \
  --version 0.34.0 --set "server.dev.enabled=true" --wait

# RSA-4096 root, RSA-4096 intermediate, a signing role locked to RSA-2048,
# and the Kubernetes auth role the cert-manager Issuer uses
./yaml/60-pki/25-vault-bootstrap.sh

# istio-csr's trust anchor is the Vault root certificate
kubectl -n vault exec vault-0 -- sh -c \
  'VAULT_TOKEN=root VAULT_ADDR=http://127.0.0.1:8200 vault read -field=certificate pki/cert/ca' > ca.pem
kubectl -n cert-manager create secret generic istio-root-ca --from-file=ca.pem=ca.pem

kubectl apply -f yaml/60-pki/20-vault-issuer.yaml
helm install cert-manager-istio-csr jetstack/cert-manager-istio-csr \
  -n cert-manager --version v0.16.0 -f yaml/60-pki/10-istio-csr-values.yaml --wait

# point istiod at the new CA, then restart it. The restart is not optional.
helm upgrade istiod oci://us-docker.pkg.dev/soloio-img/istio-helm/istiod \
  --version 1.30.3-solo -n istio-system --reuse-values \
  --set global.caAddress=cert-manager-istio-csr.cert-manager.svc:443 \
  --set pilot.env.ENABLE_CA_SERVER=false --wait
kubectl -n istio-system rollout restart deploy/istiod

istioctl -n pki-check proxy-config secret probe -o json \
  | jq -r '.dynamicActiveSecrets[] | select(.name=="default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -serial -issuer -text | head -20

kubectl apply -f yaml/20-policies-sidecar/00-peerauth-strict.yaml
kubectl apply -f yaml/10-apps-sidecar/
kubectl apply -f yaml/20-policies-sidecar/

for ns in petstore petstore-data petstore-legacy; do
  kubectl -n $ns wait --for=condition=Available deploy --all --timeout=180s
done

CHECKOUT=$(kubectl -n petstore-legacy get pod -l app=checkout -o jsonpath='{.items[0].metadata.name}')
FORTIO=$(kubectl -n petstore-legacy get pod -l app=fortio -o jsonpath='{.items[0].metadata.name}')

# 1. L7: allowed method and denied method
kubectl -n petstore-legacy exec $CHECKOUT -c checkout -- \
  curl -s -o /dev/null -w 'GET %{http_code}\n' http://catalog.petstore/
kubectl -n petstore-legacy exec $CHECKOUT -c checkout -- \
  curl -s -o /dev/null -w 'DELETE %{http_code}\n' -X DELETE http://catalog.petstore/

# 2. L4: the allowed principal
kubectl -n petstore logs deploy/data-client -c client --tail=3

# 3. L4 negative: a principal that is not allowed
kubectl -n petstore-legacy exec $FORTIO -c fortio -- \
  fortio load -c 1 -n 1 -timeout 5s -quiet tcp://redis.petstore-data:6379

# 4. the zero-downtime gate
kubectl -n petstore-legacy exec $FORTIO -c fortio -- \
  fortio load -c 4 -qps 25 -t 15s -quiet http://catalog.petstore/

helm upgrade istiod oci://us-docker.pkg.dev/soloio-img/istio-helm/istiod \
  --version 1.30.3-solo -n istio-system --reuse-values \
  --set profile=ambient --set istio_cni.enabled=true --wait

helm install istio-cni oci://us-docker.pkg.dev/soloio-img/istio-helm/cni \
  --version 1.30.3-solo -n istio-system \
  --set profile=ambient \
  --set global.hub=us-docker.pkg.dev/soloio-img/istio \
  --set global.tag=1.30.3-solo \
  --set global.platform=gke \
  --wait

helm install ztunnel oci://us-docker.pkg.dev/soloio-img/istio-helm/ztunnel \
  --version 1.30.3-solo -n istio-system \
  --set profile=ambient \
  --set hub=us-docker.pkg.dev/soloio-img/istio \
  --set tag=1.30.3-solo \
  --set global.platform=gke \
  --set env.LOG_FORMAT=json \
  --wait

./yaml/60-pki/30-vault-role.sh any

# start the load first, in another terminal
kubectl -n petstore-legacy exec $FORTIO -c fortio -- \
  fortio load -c 4 -qps 25 -t 120s -quiet http://catalog.petstore/

kubectl label ns petstore-data istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n petstore-data rollout restart deploy/redis
kubectl -n petstore-data rollout status deploy/redis

# 1. the waypoint
kubectl apply -f yaml/30-waypoints/petstore-waypoint.yaml

# 2. never enrol before this returns
kubectl -n petstore wait --for=condition=Programmed gateway/waypoint --timeout=180s

# 3. the converted policy, on the waypoint
kubectl apply -f yaml/40-policies-waypoint/10-catalog-l7-authz-targetref.yaml

# 4. route service traffic through the waypoint
kubectl label ns petstore istio.io/use-waypoint=waypoint --overwrite

# 5. remove the old selector policy BEFORE enrolment, not after
kubectl -n petstore delete authorizationpolicy catalog-get-only

# 6. enrol
kubectl label ns petstore istio.io/dataplane-mode=ambient istio-injection- --overwrite

# 7. shed the sidecars
kubectl -n petstore rollout restart deploy/catalog-v1 deploy/catalog-v2 deploy/data-client

kubectl apply -f yaml/70-northsouth/10-catalog-ingress.yaml

IP=$(kubectl -n agentgateway-system get gateway agw-ingress \
       -o jsonpath='{.status.addresses[0].value}')
curl -s -o /dev/null -w 'GET    %{http_code}\n' -H 'Host: petstore.example.com' http://$IP:8080/
curl -s -o /dev/null -w 'DELETE %{http_code}\n' -X DELETE -H 'Host: petstore.example.com' http://$IP:8080/

kubectl --context "$CONTROL" -n petstore-legacy exec $CHECKOUT -c checkout -- \
  curl -s -o /dev/null -w 'community  DELETE %{http_code}\n' -X DELETE http://catalog.petstore/
kubectl --context "$TARGET" -n petstore-legacy exec $CHECKOUT -c checkout -- \
  curl -s -o /dev/null -w 'Solo       DELETE %{http_code}\n' -X DELETE http://catalog.petstore/

# the gateway must be in the mesh to reach ambient workloads,
# and needs a PERMISSIVE carve-out because its own inbound is plaintext
kubectl label ns agentgateway-system istio.io/dataplane-mode=ambient --overwrite

# Gateway + HTTPRoute + PeerAuthentication, all three in one file
kubectl apply -f yaml/50-crosscluster/40-agentgateway-eastwest.yaml

kubectl -n agentgateway-system get svc agw-eastwest \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

helm install istio-eastwestgateway istio/gateway -n istio-gateways --create-namespace \
  -f yaml/50-crosscluster/10-eastwest-gateway-values.yaml --wait

kubectl apply -f yaml/50-crosscluster/20-exposing-side.yaml
kubectl -n istio-gateways get svc istio-eastwestgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

# hosts is a list, so this is ONE object per cluster pair, not one per call.
# Adding a service to the bridge later is a one-line edit here plus an
# HTTPRoute on the exposing side.
kubectl apply -f yaml/50-crosscluster/30-calling-side.yaml

# 1. injection back on, out of ambient. Leave use-waypoint and the
#    targetRefs policy in place so enforcement never lapses.
kubectl label ns petstore istio-injection=enabled istio.io/dataplane-mode- --overwrite

# 2. roll the pods, sidecars come back
kubectl -n petstore rollout restart deploy/catalog-v1 deploy/catalog-v2 deploy/data-client

# 3. now the selector policy can safely return
kubectl apply -f yaml/20-policies-sidecar/30-l7-authz-catalog.yaml

# 4. and only now remove the waypoint
kubectl label ns petstore istio.io/use-waypoint-
kubectl -n petstore delete authorizationpolicy catalog-get-only-waypoint
kubectl -n petstore delete gateway waypoint

What this run actually measured

Field notes

1. Measure across the cut, not after it. A 15 second fortio run after each step scores 100% and tells you nothing about the seconds in between. The 2.1% failure that exposed the ordering bug only showed up because one load generator ran across the whole sequence. Point samples cannot tell you a cutover was clean.
2. Read the waypoint's access log, not just the status code. 503 URX,UC upstream_reset_before_response_started{connection_termination} with the upstream address attached is what pointed away from the plausible theory (rolling restart drain). Two control experiments then finished the wrong theory off.
3. A selector policy follows the workload; a targetRefs policy follows the Service. That is where the ordering rule comes from. Once a workload is ambient, its selector-based policies are ztunnel's problem, and ztunnel cannot do L7. Audit for L7 selector policies before you enrol a namespace.
4. Check for PeerAuthentication mode: DISABLE carve-outs before you enrol. STRICT and PERMISSIVE both carry over cleanly, but there is no ambient equivalent of DISABLE: in ambient the transport is HBONE and it is always mTLS. Resolve the carve-out first. You can prove the carve-out is real with a plaintext request from a genuinely non-mesh pod, which is the only check that answers the question. Config dumps do not: istioctl x describe pod fails outright against ASM's older control plane, and proxy-config listener comes back empty for both a DISABLE workload and a STRICT one, because modern sidecars consolidate inbound traffic through one virtual listener.
5. In-mesh tests cannot see the north-south hole. Every caller inside the mesh is enforced correctly whether or not istio.io/ingress-use-waypoint is set, so a migration can pass every test you wrote and still be serving unenforced traffic to the internet. One curl from outside catches it. Add it to the per-namespace gate rather than trusting the in-cluster result.
6. Use the gateway you already run, where you can. The new cluster does not need a second gateway stack for east-west: the Enterprise agentgateway serving north-south takes another Gateway on the same class, with the internal-LB annotation via spec.infrastructure.annotations so the Service provisions internal on the first apply. The ASM side is the exception, and for a concrete reason: no Gateway API CRDs, and an Istio gateway there is configured by the managed control plane that is already running, so nothing new gets installed.
7. A gateway is not a remote cluster. The east-west design works because a gateway talks to its own local control plane over xDS, the same relationship every sidecar already has. Nothing in it watches the other cluster's Kubernetes API, so the CRD skew that breaks remote secrets never applies.
8. The identity tradeoff on the bridge. This terminates and re-originates mTLS at the gateway rather than carrying the original caller's identity end to end. The identity-preserving alternative, AUTO_PASSTHROUGH SNI routing, only kicks in when the control plane has no known direct address for an endpoint, which is precisely how automatic remote-secret discovery forces traffic through a gateway. A hand-written ServiceEntry always carries a real address, so it connects directly and the gateway-redirect path never activates. That makes the plain HTTP pattern the right tool for a static, migration-window reference, but the identity tradeoff is one to agree up front.

Migration checklist

1. Before you build anything
2. Read the fleet's fleetDefaultMemberConfig.mesh.management. If it is MANAGEMENT_AUTOMATIC, keep the target cluster out of the fleet.
3. List the CRD versions the ASM cluster's API server actually serves, per group. Do not assume they all lag equally.
4. Confirm no pod CIDR overlap across every cluster that will talk to another.
5. Find every PeerAuthentication with mode: DISABLE and decide what happens to it, because ambient has no equivalent.
6. Find every L7 AuthorizationPolicy that uses selector. Each one needs converting to targetRefs and deleting in the right order.
7. Per new cluster, in order
8. Gateway API CRDs first, before any Helm chart.
9. Mesh control plane in sidecar mode, and prove the workloads run on it before changing the dataplane.
10. The real ingress, first time. Note that the agentgateway licence value path is licensing.licenseKey.
11. CA work next if the mesh CA is external, and restart istiod immediately after pointing it at a new CA.
12. global.platform=gke on cni and ztunnel, from the first attempt.
13. Open the CA signing role to key_type=any before the first pod is enrolled into ambient.
14. Per namespace
15. Attach a load generator before the change and read it after. A namespace is done when its pass rate is 100%, or when every failure is explained and bounded.
16. No L7 policy: two labels and a rolling restart. No waypoint.
17. Any L7 policy: waypoint, Programmed: True, converted policy, use-waypoint, delete the old selector policy, enrol, roll.
18. Verify the workload lost its sidecar by reading both containers and initContainers in the full pod JSON.
19. Re-run the negative test as well as the happy path. A policy that stops denying is a worse outcome than one that stops allowing.
20. Test the denied case from outside the cluster, not only from a pod. A waypoint's L7 policy does not apply to ingress traffic until the Service carries istio.io/ingress-use-waypoint, and in-mesh tests pass either way.
21. Confirm enrolling this namespace disturbed nothing in the others: no unexpected restarts anywhere else.
22. Drain, then decommission
23. Bring the service up on the new cluster and prove it there before any client is moved: same GET and DELETE results, same load-generator pass rate, same negative test as the ASM-side baseline.
24. Move clients team by team, not in one cut. A namespace is drained when the ASM-side load generator reads zero and the new cluster's reads the full volume, with no gap and no double-count in between.
25. Watch both sides through each move. Two load generators, one per cluster, both running across the cutover.
26. Scale the ASM-side Deployments to zero and leave them there for a full business cycle before deleting anything. A scaled-to-zero Deployment is a rollback that takes seconds; a deleted namespace is a redeploy.
27. While they sit at zero, confirm nothing still calls them: no traffic in the ASM-side access logs, no DNS lookups for the old hostnames, and no ServiceEntry anywhere still pointing at them.
28. Check what else referenced that service before you delete it: CI jobs, cron workloads, dashboards, alerts and any hard-coded cluster-local hostname in another team's config.
29. Delete the app namespaces on the ASM side only once the scaled-to-zero period has passed cleanly.
30. Remove the migration scaffolding last and in this order: the ServiceEntry resources on each calling side, then the Gateway and VirtualService, then the east-west gateway release and its internal LoadBalancer. Removing the gateway first breaks any call you missed, and breaks it silently.
31. Confirm the internal LoadBalancers are actually gone from the project. A deleted Service with a stuck finalizer leaves the forwarding rule and the bill behind.
32. Before you call it done
33. Check the access-log shape. ztunnel's JSON L4 log is structurally different from an Envoy sidecar log, so any SIEM rule written against the old format silently stops matching.
34. Re-check any canary or subset weighting for callers that arrive over the east-west bridge. They do not get the in-mesh VirtualService.
35. Keep the signing role on any until the last sidecar in the estate is gone.
36. Point north-south client traffic at the new cluster's external ingress and retire the old one only after the drain is complete.

Why the Enterprise distribution, on this specific migration

• The mixed period is the whole migration, and that is where the interop matters. A large estate is mixed for months. On community images a caller that still has a sidecar does not route through the destination's waypoint, so the waypoint's L7 policy does not apply to it, which was measured directly here: DELETE 200 on community, DELETE 403 on Solo, same caller, same manifests. Every namespace that migrates before its callers do has that hole until the callers follow.
• L7 telemetry without a waypoint. The L4-only namespace in this lab loses request-level visibility on community images the moment it is enrolled, because ztunnel is L4. The Solo ztunnel emits HTTP-level telemetry with no waypoint at all, so a namespace like petstore-data keeps its request metrics after migrating.
• Sharper identity at L4. Upstream ztunnel authorises on the ServiceAccount identity, so two workloads sharing a ServiceAccount are indistinguishable to an L4 policy. The Solo images add workload claims: pod-level attributes carried in the certificate that ztunnel can authorise on with no waypoint. Certificates as workload identity in ambient runs that end to end.
• A support window that matches how long migrations take. Community minors leave support quickly. Solo maintains an N-4 patched window with CVE backports, so the version you started the migration on is still patched when the last team finishes.

See also

• Sidecar to Ambient Upgrade: the same migration on one kind cluster, installed by the Gloo Operator, with the subset canary moved to an HTTPRoute and a single-label rollback.
• Sidecar to Ambient on OSS Istio: the fully upstream version, with one cluster-wide waypoint instead of one per namespace.
• Vault + istio-csr: sidecars on RSA, ambient on EC: the CA transition in this lab, in depth, on kind.
• Certificates as workload identity in ambient: the L4 identity model the Redis policy here relies on.

Versions

Built and verified on both editions: