Whose Waypoint? L7 across an Ambient Mesh, by Tom O'Rourke

Overview

You have two clusters. The same app, reviews, is deployed in both — for resilience, blast-radius isolation, data residency, or just because the platform team grew that way. A pod in cluster-east needs to call reviews and shouldn't care which cluster the answering pod actually runs in. The security team needs one set of rules — "only signed-in users can GET on this path" — applied consistently, no matter which side of the WAN serves the request. And it all has to survive an entire cluster going dark.

You could solve this north-south: a global load balancer in front of two clusters, DNS-routed by geography, public hostnames per cluster, AuthZ re-enforced at each cluster's ingress. That works for human users at the edge. It falls apart for east-west service-to-service calls inside the mesh — the caller's identity is lost at the LB, every cross-cluster hop is an opaque public hostname, and the L7 rule has to be duplicated and kept in sync at the ingress of every cluster in the fleet.

Federation collapses that. The control planes peer, endpoints are discovered both ways, calls go directly over HBONE on the data plane, SPIFFE identities are preserved end to end, and the L7 policy lives once — in the caller's mesh — and applies regardless of which cluster the chosen endpoint lives in. The sameness-driven pattern in this article — a single label that promotes a Service to a logical <svc>.<ns>.mesh.internal hostname spanning every peered cluster, with the caller-side waypoint enforcing L7 across the fleet — is Solo Enterprise for Istio. You label a Service in two clusters with solo.io/service-scope: global, the control plane builds the synthetic logical service, and every workload that calls that hostname can reach endpoints in either cluster. Upstream OSS Istio supports multi-cluster ambient at the lower-level primitive layer (east-west gateways, shared trust, remote secrets), but doesn't ship a turn-key equivalent of the solo.io/service-scope=global model.

The interesting question is where L7 enforcement happens. ztunnel is L4 only — by design, it cannot evaluate HTTP method or path and will fail closed if asked. So somebody has to run a waypoint. The answer — and the part that surprises people coming from the upstream OSS Istio mental model — is that the caller's waypoint enforces, not the destination's. The destination's waypoint is not on the path when traffic arrives via the east-west gateway.

Two clusters, both primaries

Solo's multicluster peering pattern is multi-primary: every cluster runs its own istiod, every cluster shares a root CA so SPIFFE identities verify both ways, and every cluster installs the same ambient stack — ztunnel DaemonSet, an east-west gateway on :15008, and a waypoint in any namespace that needs L7. The picture below is the topology you have once IstioLifecycleManager, GatewayLifecycleManager and RootTrustPolicy have reconciled across both clusters. The flow diagram further down then traces one request across this same topology.

istiod / shared trust waypoint (L7) ztunnel (L4) east-west gateway workload pod

Symmetric, not hub-and-spoke. Both clusters run a full istiod and a full ambient data plane. The RootTrustPolicy shares a CA so SPIFFE identities verify both ways; the istio-remote-secret in each cluster lets each istiod see the other's endpoints. The logical global service at the top (reviews.bookinfo.mesh.internal) is what every Solo sameness-labelled reviews Service folds into — clients call that hostname and reach a pod in either cluster. Which waypoint runs the L7 policy depends entirely on where the caller lives.

The path, end to end

pod waypoint (L7) ztunnel (L4) HBONE tunnel plain TCP

Read it like this. The cart pod calls the global service reviews.bookinfo.mesh.internal. ztunnel-east sees the namespace has istio.io/use-waypoint=waypoint and steers the connection into waypoint-east first. That waypoint is where the L7 AuthorizationPolicy runs — method, path, headers. Only after that decision does the request continue: ztunnel-east wraps it in HBONE, the east-west gateway in cluster-west forwards on SNI alone, and ztunnel-west delivers a plain TCP segment to the reviews pod. The destination's own waypoint is bypassed — that one only matters for callers inside cluster-west.

FAQ

Q1 Does the caller's waypoint only run if there's a local pod for the service?

No. The caller's waypoint runs whenever the namespace (or service) is labelled istio.io/use-waypoint=waypoint and the destination is a global service the caller can resolve. It does not matter whether a backing pod exists in the local cluster. The local pod's presence is not the trigger — the caller's location is.

If the caller is in cluster-east and you've put a waypoint in front of reviews there, the policy fires for every call from cluster-east to reviews.bookinfo.mesh.internal, including the calls that land on a remote pod in cluster-west.

Q2 When the request reaches cluster-west, does the destination's waypoint run again?

No. Cross-cluster ingress is e-w gateway → ztunnel-west → pod. The destination's waypoint is not on that path. The east-west gateway is an SNI router on :15008 — it forwards the inner HBONE tunnel and never sees a waypoint. ztunnel-west then verifies the peer's SPIFFE identity from the inner mTLS, optionally enforces an L4 AuthorizationPolicy on principals: / ports, and delivers plain TCP to the pod.

So L7 is enforced exactly once, in cluster-east. The destination cluster's waypoint matters only for traffic originating inside cluster-west.

Q3 So is the destination unprotected?

Not at L4. The inner HBONE tunnel preserves the caller's SPIFFE identity all the way to ztunnel-west — the east-west gateway terminates only the outer TLS. That means L4 AuthorizationPolicy rules at the destination (match on principals:, ports, namespaces) still evaluate correctly against the caller's actual identity. What you cannot do at the destination is enforce L7 — for that, you need a waypoint, and the only waypoint on the path is in cluster-east.

The CRDs behind the flow

Three resources are doing the work. A label on the Service turns it into a global service. A Gateway API Gateway in the caller's namespace provisions the waypoint. An AuthorizationPolicy targeting that Service tells the waypoint what to allow at L7.

🛰️ Global Service · sameness label v1 Service

Apply the same label to the same-named Service in every cluster you want to be part of the global pool. The control plane stitches them into one synthetic logical service reachable at <svc>.<ns>.mesh.internal. Clients must use that hostname — svc.cluster.local stays cluster-local and won't pick up remote endpoints.

Service · marked global in both clustersapply identically in cluster-east AND cluster-west

apiVersion: v1
kind: Service
metadata:
  name: reviews
  namespace: bookinfo
  labels:
    # Solo sameness label — promotes the Service into a logical
    # global service reachable as reviews.bookinfo.mesh.internal
    solo.io/service-scope: global
spec:
  ports:
    - name: http
      port: 9080
      targetPort: 9080
  selector:
    app: reviews

🛣️ Waypoint Gateway gateway.networking.k8s.io/v1

The waypoint is just a Gateway with class istio-waypoint listening on :15008 for protocol HBONE. The namespace label istio.io/use-waypoint=waypoint is what makes ztunnel steer connections from any pod in the namespace through this waypoint before they leave the cluster. Apply it in the caller's namespace.

Gateway + Namespace · waypoint + steering labelapplied in cluster-east, caller's namespace

# Namespace-scoped waypoint proxy. ztunnel sees the use-waypoint label
# and hands HBONE traffic off to this Gateway, where Envoy runs L7
# filters before forwarding to the destination workload.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: bookinfo
spec:
  gatewayClassName: istio-waypoint
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
---
# This is the line that activates the waypoint for the whole namespace.
# Anything sent to a Service in this namespace gets steered through the
# waypoint by ztunnel — no per-app config needed.
apiVersion: v1
kind: Namespace
metadata:
  name: bookinfo
  labels:
    istio.io/dataplane-mode: ambient
    topology.istio.io/network: network-east
    istio.io/use-waypoint: waypoint

🔐 AuthorizationPolicy · L7 ALLOW rule security.istio.io/v1

Attached to the global Service via targetRefs. The waypoint evaluates this on every request — method, path, headers. Because the rule binds to the Service (not the workload), it fires for callers regardless of which cluster the destination pod actually lives in. It only works because a waypoint is in front of the Service; without one, ztunnel cannot see HTTP and the policy would silently match nothing.

AuthorizationPolicy · GET-only allow-list on the Serviceapplied in cluster-east, same namespace as the Service

# L7 enforcement that's only possible *because* a waypoint is in front
# of the Service. Without a waypoint, ztunnel could only enforce L4
# (peer identity); with the waypoint it can see HTTP method + path
# and block anything that isn't a GET on the allow-listed paths.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: reviews-read-only
  namespace: bookinfo
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: reviews
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET"]
            paths: ["/reviews/*"]

What goes wrong (and how it looks)

Four traps that bite people moving from single-cluster ambient to a peered, sameness-style deployment.

1. Client uses svc.cluster.local instead of mesh.internal. The client resolves the local Service, which only has local endpoints. Cross-cluster traffic never happens, failover to the remote cluster doesn't fire, and the L7 policy still runs — but it's a single-cluster story dressed up as a multi-cluster one. The fix is to use the <svc>.<ns>.mesh.internal hostname for every call you want to span clusters.

2. Mixed-mode (sidecar caller, ambient destination) under upstream OSS Istio. In community Istio, sidecar-enabled or Envoy-based workloads cannot utilise waypoint proxies — the sidecar dials the destination directly and the destination's waypoint is bypassed. Solo Enterprise for Istio patches this: the sidecar's outbound traffic still goes through the waypoint, so the L7 policy fires consistently. If you're testing on upstream OSS and the policy doesn't apply, that's the cause.

3. AuthorizationPolicy placed in the wrong namespace. targetRefs on an AuthorizationPolicy does not support cross-namespace references — the policy and the Service it targets must live in the same namespace. Put the policy where the Service is, in the cluster whose waypoint will actually enforce it (the caller's cluster). A policy in cluster-west's copy of the namespace will only fire for callers inside cluster-west.

4. Expecting telemetry to "look the same" on both sides. L7 metrics (HTTP code, method, path) only exist where the waypoint ran — i.e. on the caller's cluster. The destination cluster sees L4 connection metrics from ztunnel-west and that's it. When you build dashboards for a global service, aggregate by caller cluster and accept that "5xx by destination cluster" isn't a meaningful slice from the destination's metrics alone.

Components on the path

One row per box the request touches, in order. The L4 / L7 column is which layer each component is allowed to enforce on this path — useful when sketching where a new rule should live.

Hop	Component	Where it runs	What it enforces	L4 / L7
—	`cart pod` pod	cluster-east · bookinfo	Sends an HTTP request to `reviews.bookinfo.mesh.internal:9080`. No proxy, no SDK.	— / —
1	`ztunnel-east` ztunnel	cluster-east · DaemonSet · per node	CNI redirect catches it, sees `use-waypoint`, steers into the waypoint.	L4 only
2	`waypoint-east` waypoint	cluster-east · bookinfo · Gateway istio-waypoint	L7 AuthorizationPolicy fires here. Method, path, headers, peer SPIFFE.	L4 + L7
3	`ztunnel-east` ztunnel	cluster-east · same DaemonSet	Wraps in HBONE, dials the remote E-W gateway on :15008.	L4 only
4	`istio-eastwestgateway`	cluster-west · LoadBalancer :15008	SNI router. Forwards inner HBONE. Never sees the L7 request.	— (transport)
5	`ztunnel-west` ztunnel	cluster-west · DaemonSet · per node	Verifies caller's SPIFFE from inner mTLS. L4 AuthorizationPolicy on `principals:` still works.	L4 only
—	`reviews pod` pod	cluster-west · bookinfo	Sees plain TCP from ztunnel-west's pod IP. Identity already authenticated upstream.	— / —
—	`waypoint-west` waypoint	cluster-west · bookinfo · NOT on this path	Only fires for callers inside cluster-west. Bypassed for cross-cluster ingress.	— (bypassed)

Where to go from here

The packet-level walkthrough of the HBONE leg sitting in the middle of this diagram is on the HBONE + east-west page — same colours, same metaphors. The full set of ambient CRDs that wire all of this together — Gateway, PeerAuthentication, AuthorizationPolicy, ServiceEntry — are grouped on the Istio Ambient CRDs visual map. For how the east-west gateway gets installed across N clusters in the first place, see Gloo Operator across N clusters.

The customer value of this pattern — one policy decision per request, regardless of where the destination pod actually lives, enforced consistently across clusters — is what underpins compliance frameworks like ISO 27001's segregation-of-duties controls and DORA Article 28's ICT third-party risk requirements, where "who called what, with which identity, against which rule" has to be answerable at a single point.