Cloud Connectivity Lab — failover, waypoint, egress

Prerequisite — deploy Bookinfo + the agentgateway ingress

The standup lab installs the platform only — no test workloads. This lab's demos all use Bookinfo as the test app, so deploy it on both clusters, label its namespace ambient, mark productpage as a global multicluster service, and apply the bookinfo-gateway Gateway + HTTPRoute on east.

Deploy Bookinfo to both clusters

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX create namespace bookinfo 2>/dev/null || true
  kubectl --context $CTX label namespace bookinfo \
    istio.io/dataplane-mode=ambient \
    topology.istio.io/network=${CTX#kind-} --overwrite

  BOOKINFO=https://raw.githubusercontent.com/istio/istio/release-1.24/samples/bookinfo/platform/kube
  kubectl --context $CTX apply -n bookinfo -f $BOOKINFO/bookinfo.yaml
  kubectl --context $CTX apply -n bookinfo -f $BOOKINFO/bookinfo-versions.yaml

  kubectl --context $CTX -n bookinfo wait \
    --for=condition=Ready pod -l app=productpage --timeout=180s
done

Label productpage as a global multicluster service

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX label svc productpage -n bookinfo \
    solo.io/service-scope=global --overwrite
done

# Confirm: istioctl multicluster check should now show "1 globally shared service"
istioctl --context $CLUSTER1 multicluster check | grep "Shared Services"

Apply the agentgateway ingress (Gateway + HTTPRoute) on east

kubectl --context $CLUSTER1 apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  gatewayClassName: enterprise-agentgateway
  listeners:
  - name: http
    port: 8080
    protocol: HTTP
    allowedRoutes: { namespaces: { from: Same } }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: productpage
  namespace: bookinfo
spec:
  parentRefs:
  - name: bookinfo-gateway
  rules:
  - backendRefs:
    - kind: Hostname               # Solo Gateway-API extension
      group: networking.istio.io
      name: productpage.bookinfo.mesh.internal
      port: 9080
EOF

Wait for the controller to create + ready the backing pod

# 1. Wait for the Gateway itself to be Programmed (controller has reconciled)
kubectl --context $CLUSTER1 -n bookinfo wait \
  --for=condition=Programmed gateway/bookinfo-gateway --timeout=60s

# 2. Now the backing pod exists — wait for it to be Ready
kubectl --context $CLUSTER1 -n bookinfo wait \
  --for=condition=Ready pod -l gateway.networking.k8s.io/gateway-name=bookinfo-gateway \
  --timeout=120s

Verify — port-forward and curl through the ingress

kubectl --context $CLUSTER1 -n bookinfo port-forward svc/bookinfo-gateway 8080:8080 &
curl -s -o /dev/null -w "ingress: HTTP %{http_code}\n" http://localhost:8080/productpage
kill %1
# → ingress: HTTP 200

Failover — mesh-layer cross-cluster failover

This step demonstrates Solo Istio's global service failover at the mesh layer. When productpage-v1 is scaled to zero on east, calls to the global hostname productpage.bookinfo.mesh.internal transparently route to west via the east-west HBONE gateway.

Scale productpage down on east

kubectl --context $CLUSTER1 scale deploy productpage-v1 -n bookinfo --replicas=0
kubectl --context $CLUSTER1 -n bookinfo wait --for=delete pod -l app=productpage --timeout=30s

Confirm there are no local endpoints, but the global VIP is healthy

istioctl --context $CLUSTER1 ztunnel-config service | grep productpage
# bookinfo  autogen.bookinfo.productpage  240.240.0.2,2001:2::2  None  1/1   ← cross-cluster
# bookinfo  productpage                   10.96.x.x              None  0/0   ← local svc, no eps
# bookinfo  productpage-v1                10.96.x.x              None  0/0

Curl through the mesh using the global hostname

Run a curl pod inside the ambient mesh so its outbound call is intercepted by ztunnel. The synthetic VIP 240.240.0.2 resolves to the global hostname, which istiod has programmed with the remote (west) endpoint.

kubectl --context $CLUSTER1 -n bookinfo run mesh-curl \
  --rm -i --restart=Never --image=curlimages/curl:8.5.0 -- \
  curl -sS -o /dev/null -w "HTTP %{http_code} via %{remote_ip}\n" \
    http://productpage.bookinfo.mesh.internal:9080/productpage
# → HTTP 200 via 240.240.0.2

Compare — calling the regular short hostname does NOT fail over

This is by design: the regular Service VIP only includes local endpoints. Locality-aware failover only kicks in for the *.mesh.internal global hostname.

kubectl --context $CLUSTER1 -n bookinfo run mesh-curl \
  --rm -i --restart=Never --image=curlimages/curl:8.5.0 -- \
  curl -sS -o /dev/null -w "HTTP %{http_code} via %{remote_ip}\n" \
    http://productpage:9080/productpage
# → HTTP 000 (connection reset — local VIP 10.96.x.x has 0 endpoints)

Scale productpage back up on east

kubectl --context $CLUSTER1 scale deploy productpage-v1 -n bookinfo --replicas=1
kubectl --context $CLUSTER1 -n bookinfo wait --for=condition=Ready pod -l app=productpage --timeout=60s

L7 policy — enterprise-agentgateway-waypoint

Deploy an AgentGateway waypoint proxy for the bookinfo namespace, then use an HTTPRoute to pin reviews traffic to a specific version. ztunnel transparently routes all bookinfo traffic through the waypoint for L7 processing.

Create EnterpriseAgentgatewayParameters and waypoint Gateway

kubectl --context $CLUSTER1 apply -f - <<'EOF'
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
  name: waypoint-params
  namespace: bookinfo
spec:
  istioClusterId: east-ag        # must match istiod --clusterID
  ca:
    trustDomain: cluster.local   # must match mesh trustDomain
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agw-waypoint
  namespace: bookinfo
spec:
  gatewayClassName: enterprise-agentgateway-waypoint
  listeners:
    - name: proxy
      port: 15008
      protocol: HBONE
  infrastructure:
    parametersRef:
      group: enterpriseagentgateway.solo.io
      kind: EnterpriseAgentgatewayParameters
      name: waypoint-params
EOF

Label the namespace and verify ztunnel sees the waypoint

kubectl --context $CLUSTER1 label namespace bookinfo \
  istio.io/use-waypoint=agw-waypoint --overwrite

# Wait ~5s, then verify all bookinfo services show "agw-waypoint"
istioctl --context $CLUSTER1 ztunnel-config service | grep bookinfo
# → bookinfo   reviews   10.x.x.x   agw-waypoint   3/3

Apply header-based reviews routing

Sends user jason to reviews-v2 (black stars); everyone else gets reviews-v1 (no stars). The Service parentRef is the Istio Ambient waypoint pattern.

kubectl --context $CLUSTER1 apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: reviews
  namespace: bookinfo
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: reviews
    port: 9080
  rules:
  - matches:
    - headers:
      - name: end-user
        value: jason
    backendRefs:
    - name: reviews-v2
      port: 9080
  - backendRefs:
    - name: reviews-v1
      port: 9080
EOF

Test — default user (no stars, reviews-v1)

kubectl --context $CLUSTER1 -n bookinfo \
  port-forward svc/bookinfo-gateway 8080:8080 &

curl -s http://localhost:8080/productpage | grep "glyphicon-star"
# → no output — locked to reviews-v1 (no ratings)

kill %1

Test — user jason via browser (black stars, reviews-v2)

Open http://localhost:8080/productpage in a browser and log in as jason / jason. productpage sets a session cookie with end-user: jason which the waypoint sees on the internal call to reviews.

Confirm traffic went through the agentgateway waypoint

kubectl --context $CLUSTER1 -n bookinfo \
  logs -l gateway.networking.k8s.io/gateway-name=agw-waypoint \
  --tail=5
# → gateway=bookinfo/agw-waypoint listener=waypoint route=bookinfo/reviews http.status=200

Clean up routing

kubectl --context $CLUSTER1 delete httproute reviews -n bookinfo
kubectl --context $CLUSTER1 delete gateway agw-waypoint -n bookinfo
kubectl --context $CLUSTER1 label namespace bookinfo \
  istio.io/use-waypoint- --overwrite

Egress gateway — enterprise-agentgateway-waypoint

Controls which workloads can reach external services using an enterprise-agentgateway-waypoint in a dedicated egress namespace.

Create namespace and deploy egress gateway

kubectl --context $CLUSTER1 create namespace istio-egress

kubectl --context $CLUSTER1 apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: egress-gateway
  namespace: istio-egress
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: enterprise-agentgateway-waypoint
  listeners:
  - name: mesh
    port: 15088
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
EOF

kubectl --context $CLUSTER1 label ns istio-egress \
  istio.io/use-waypoint=egress-gateway

Define httpbin.org as an external service and apply AuthorizationPolicy

kubectl --context $CLUSTER1 apply -f - <<'EOF'
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: httpbin.org
  namespace: bookinfo
  labels:
    istio.io/use-waypoint: egress-gateway
    istio.io/use-waypoint-namespace: istio-egress
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: httpbin.org-tls
  namespace: bookinfo
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ratings-to-httpbin
  namespace: bookinfo
spec:
  targetRefs:
  - kind: ServiceEntry
    group: networking.istio.io
    name: httpbin.org
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
        - "cluster.local/ns/bookinfo/sa/bookinfo-ratings"
EOF

Test — ratings should succeed

kubectl --context $CLUSTER1 exec -it \
  $(kubectl --context $CLUSTER1 get pod -l app=ratings -n bookinfo \
    -o jsonpath='{.items[0].metadata.name}') \
  -n bookinfo -- curl -s httpbin.org/get | head -5

Test — reviews should be blocked

kubectl --context $CLUSTER1 exec -it \
  $(kubectl --context $CLUSTER1 get pod -l app=reviews -n bookinfo \
    -o jsonpath='{.items[0].metadata.name}') \
  -n bookinfo -- curl -sv httpbin.org/get 2>&1 | grep -E "RBAC|403|denied"