Multi-region load balancing and failover on Solo ambient

The architecture

Two clusters, each self-sufficient. There is a management/telemetry cluster in the customer's target picture (Solo UI, telemetry gateway) but it is not in the request path and not in the discovery path — peering is direct istiod-to-istiod. That is the single most important fact for a bank running tens of clusters with no central coordination: nothing you deploy here can take traffic down by failing.

Stand it up

Five scripts. Everything creates paid AWS infrastructure, so the profile is forced explicit (LAB_AWS_PROFILE) and there is a teardown that removes it all in the right order.

# two EKS clusters, one per region (m5.large x3, headroom to 12 for the scale demo)
eksctl create cluster -f eksctl/mesh-eu-central.yaml   # eu-central-1
eksctl create cluster -f eksctl/mesh-eu-west.yaml      # eu-west-1

export LAB_AWS_PROFILE=<your-aws-profile>     # forced explicit — never inherits a sourced AWS_PROFILE
export SECRETS_FILE=~/code/solo/secrets/secrets-envs.sh   # SOLO_ISTIO_LICENSE_KEY

./scripts/01-istio.sh      # Solo ambient on both, plain Helm, shared root CA
./scripts/02-peering.sh    # east-west gateways on NLBs + istiod peering + remote secrets
./scripts/03-app.sh        # region-echo as a global service in both regions

Failover at the mesh layer

This is the layer the customer's Bookinfo demo showed, done cleanly. The client calls region-echo.shop.svc.cluster.local and never changes. With trafficDistribution: PreferClose istiod translates the service to ztunnel's Failover load-balancing policy — routing preference Network → Region → Zone, healthy endpoints only. Traffic stays local; it crosses regions only when the local endpoints are gone.

# the service that makes it work — a k8s-native global service, locality-preferred
kind: Service
metadata:
  name: region-echo
  namespace: shop
  labels:
    app: region-echo
    istio.io/global: "true"          # published across the peered mesh
spec:
  trafficDistribution: PreferClose   # -> ztunnel Failover LB (Network,Region,Zone)
  ports: [{ name: http, port: 8080, targetPort: 8080, appProtocol: http }]

# the demo: scale the local endpoints to 0, watch the client, scale back
kubectl -n shop scale deploy/region-echo --replicas=0    # eu-central only

Phase 1 — steady state
  -> {"region": "eu-central-1", "pod": "region-echo-7c6dc74977-8bgsd"}   <- local

Phase 2 — eu-central endpoints scaled to 0
  -> {"region": "eu-west-1", "pod": "region-echo-597c98cb47-vfvgf"}      <- cross-region, over the east-west GW

Phase 3 — restored
  -> {"region": "eu-central-1", "pod": "region-echo-7c6dc74977-6dvfp"}   <- local again

Failover at the edge layer — pick one

The mesh layer (above) handles callers already inside the mesh. External clients hitting a regional endpoint need a front door that fails a whole region out, and AWS gives you two independent ways to build it. They are alternatives, not layers — you pick one. The choice comes down to a single trade-off: what the client holds onto, and therefore how fast the cutover is.

Both are built and demonstrated live below, each end to end.

Edge approach A · Route 53 (DNS failover)

A Route 53 failover record set over the two regional ingress NLBs. The client resolves the hostname and gets the primary region's address; when the primary's health check fails, Route 53 hands out the secondary region's address on the next resolution. Simple, no extra service to run — the cutover is bounded by the record TTL and whatever the client and its resolvers cache.

What you create

• A hosted zone you control (a domain or subdomain).
• Two health checks — TCP :80 against each regional ingress NLB.
• One failover record set — a PRIMARY and a SECONDARY record for the same name, each tied to its region's health check, with a low TTL.

How — the exact commands

# 1. one health check per region, TCP:80 against that region's ingress NLB
HC1=$(aws route53 create-health-check --caller-reference eu-central-$(date +%s) \
  --health-check-config Type=TCP,FullyQualifiedDomainName=$ING1_NLB,Port=80,RequestInterval=10,FailureThreshold=2 \
  --query 'HealthCheck.Id' --output text)
HC2=$(aws route53 create-health-check --caller-reference eu-west-$(date +%s) \
  --health-check-config Type=TCP,FullyQualifiedDomainName=$ING2_NLB,Port=80,RequestInterval=10,FailureThreshold=2 \
  --query 'HealthCheck.Id' --output text)

# 2. the failover record set: PRIMARY -> eu-central, SECONDARY -> eu-west,
#    each bound to its health check, TTL 15s
aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch '{
  "Changes": [
    {"Action":"UPSERT","ResourceRecordSet":{
      "Name":"region-echo.example.com","Type":"CNAME","TTL":15,
      "SetIdentifier":"primary","Failover":"PRIMARY",
      "HealthCheckId":"'$HC1'","ResourceRecords":[{"Value":"'$ING1_NLB'"}]}},
    {"Action":"UPSERT","ResourceRecordSet":{
      "Name":"region-echo.example.com","Type":"CNAME","TTL":15,
      "SetIdentifier":"secondary","Failover":"SECONDARY",
      "HealthCheckId":"'$HC2'","ResourceRecords":[{"Value":"'$ING2_NLB'"}]}}
  ]}'

kubectl --context $CTX1 -n kgateway-system scale deploy/ingress --replicas=0   # primary down
dig +short region-echo.example.com     # watch it switch to the eu-west NLB
curl -s http://region-echo.example.com/

Phase 1 — steady state (primary = eu-central-1)
  serves:      "region": "eu-central-1"

Phase 2 — eu-central ingress down (primary health check fails)
  flipped:     "region": "eu-west-1"
  resolves to: a3ddd3a7336194d77af4f662963f25f5-...elb.eu-west-1.amazonaws.com   <- the secondary NLB

Phase 3 — restore
  primary returns once its health check passes again and the TTL expires

Edge approach B · Global Accelerator (anycast)

Two static anycast IPs sit in front of both regional NLBs. The client always targets the same IPs; Global Accelerator health-checks each region and routes to the nearest healthy one, rerouting internally on failure with no DNS involved. Nothing waits on a resolver cache — this is the answer when recovery time is the priority.

What you create

• One accelerator — it hands you two static anycast IPs (and a DNS name) as the front door.
• One TCP :80 listener.
• Two endpoint groups — one per region, each pointing at that region's ingress NLB ARN, with a health check. No hosted zone, no domain.

How — the exact commands

# 1. the accelerator (returns 2 static anycast IPs + a DNS name)
GA_ARN=$(aws globalaccelerator create-accelerator --name mesh-multiregion \
  --region us-west-2 --query 'Accelerator.AcceleratorArn' --output text)

# 2. a TCP:80 listener
LARN=$(aws globalaccelerator create-listener --accelerator-arn $GA_ARN \
  --protocol TCP --port-ranges FromPort=80,ToPort=80 \
  --region us-west-2 --query 'Listener.ListenerArn' --output text)

# 3. one endpoint group per region, pointing at that region's ingress NLB ARN
aws globalaccelerator create-endpoint-group --listener-arn $LARN \
  --endpoint-group-region eu-central-1 \
  --endpoint-configurations EndpointId=$ING1_NLB_ARN,Weight=100 \
  --health-check-port 80 --health-check-protocol TCP \
  --health-check-interval-seconds 10 --threshold-count 2 --region us-west-2
# (repeat for eu-west-1 -> $ING2_NLB_ARN)

# take down the region GA is actually serving THIS client, watch the same anycast address
kubectl --context $CTX -n kgateway-system scale deploy/ingress --replicas=0
for i in $(seq 20); do curl -s http://$GA_DNS/; done   # count the regions

Phase 1 — steady state
  15  "region": "eu-central-1"
   5  "region": "eu-west-1"

Phase 2 — eu-west ingress down (the region GA was serving us)
  flipped to eu-central-1 after 40s
  19  "region": "eu-central-1"
   1  "region": "(timeout)"        <- single request mid-cutover

Phase 3 — restored
  20  "region": "eu-central-1"

Does it prefer the closest region? (yes, and there is no latency check)

A direct answer to the second requirement. Locality-weighted routing in the mesh is deterministic, not measured. trafficDistribution: PreferClose becomes a ztunnel Failover policy whose routing preference is Network → Region → Zone over healthy endpoints only. "Closest" is read from topology labels the node already carries — there is no latency probe, and there does not need to be. Latency-based steering, if you want it, is the client-side job (Route53 latency records or Global Accelerator), not the mesh's.

$ istioctl ztunnel-config service <ztunnel> -o json | jq '.[] | select(.name=="region-echo").loadBalancer'
{
  "mode": "Failover",
  "routingPreferences": ["Network", "Region", "Zone"],
  "healthPolicy": "OnlyHealthy"
}

Scale to 1000 tenants — test the thing that actually scales

The third requirement asked whether "the Gloo Management Plane can sync discovery for 1k tenants without hitting rate limits." The honest reframing: in ambient peering there is no management plane in the discovery path. Peering is direct istiod-to-istiod; the management/telemetry cluster carries the UI and metrics only and cannot take traffic down. So the component to load-test is istiod (per cluster) and the peering fan-out — which is what the scale script measures.

./scripts/06-scale.sh 100      # ramp to 100 tenants on both clusters (default)
./scripts/06-scale.sh 1000     # the full number — scale the nodegroups first:
#   eksctl scale nodegroup --cluster mesh-eu-central -r eu-central-1 --name workers -N 10

# each tenant = one namespace + one global service (http-echo). The script captures:
#   - pilot_xds_pushes / convergence time from istiod at 100/500/1000
#   - istiod + ztunnel CPU/memory (kubectl top)
#   - time for a freshly-created global service to be served from the PEER cluster

Which layer for which failure

Tear it down

This is billed infrastructure — two EKS clusters, four NLBs, a Global Accelerator, Route53 health checks. The teardown removes them in the order that avoids orphaned load balancers.

export LAB_AWS_PROFILE=<your-aws-profile>
./scripts/teardown.sh    # GA -> health checks -> LB services -> both clusters

See also

• Ambient deployment and L4/L7 policies — identity, ztunnel authz, agentgateway waypoint on one kind cluster
• Segments — isolate tenants and bound the blast radius across the peered mesh
• Solo docs — Solo Enterprise for Istio (ambient, multicluster)

Versions

Built and verified on: