HBONE + east-west — what a cross-cluster Ambient packet does

The path, end to end

What happens at each hop

One block per hop on the diagram. The YAML or pseudo-config under each is the thing you'd actually touch — or grep for — to see what's happening at that point in the path.

# Conceptually — istio-cni's redirect for ambient-enabled pods.
# (Don't run this; the real rules live in the istio-cni-node DaemonSet
#  and use marks + tproxy. This is the moral equivalent.)

iptables -t mangle -A PREROUTING \
  -i pod-veth \
  -p tcp \
  -j TPROXY \
    --tproxy-mark 0x111/0xfff \
    --on-port 15001 \
    --on-ip 127.0.0.1

# Result: every outbound TCP segment from app=cart arrives at
# ztunnel's outbound socket on the local node, with the original
# 5-tuple preserved.

# What ztunnel-east actually emits on the wire (conceptually).
#
# 1. Outer TLS 1.3 ClientHello to :15008
#    SNI:                       spiffe://cluster.local/ns/payments/sa/cart
#    ALPN:                      h2
#    Client cert (mTLS):        spiffe://cluster.local/ns/payments/sa/cart
#
# 2. Inside the TLS session, a single HTTP/2 CONNECT stream:

CONNECT 10.20.7.30:8080 HTTP/2
:authority: 10.20.7.30:8080
baggage:    k8s.cluster.name=cluster-east,
            k8s.namespace.name=payments,
            k8s.pod.name=cart-7d9f-abcde
x-envoy-original-dst-host: 10.20.7.30:8080

# The original app bytes are now the body of that CONNECT stream.
# Outer TLS = transport. Inner CONNECT = "give me back my L4
# connection on the other side." mTLS validates both ends.

# What the gateway sees on an incoming connection on :15008
#
# TLS ClientHello — SNI = spiffe://cluster.local/ns/payments/sa/cart
#                   ALPN = h2
#                   Client cert principal = spiffe://...sa/cart
#
# Routing decision:
#   - SNI's trust domain (cluster.local) is in the accepted set
#   - destination addr in the inner CONNECT (10.20.7.30:8080)
#     belongs to node-west-3
#   - dial node-west-3's ztunnel pod IP : 15008
#
# Service that fronts the gateway:

apiVersion: v1
kind: Service
metadata:
  name: istio-eastwestgateway
  namespace: istio-eastwest
spec:
  type: LoadBalancer
  selector:
    app: istio-eastwestgateway
  ports:
    - name: tls
      port: 15008
      targetPort: 15008
      protocol: TCP

# From the destination pod's view (tcpdump on lo / eth0):
#
#   src = 10.20.7.1   (ztunnel-west's pod IP on node-west-3)
#   dst = 10.20.7.30:8080  (checkout pod)
#   plain TCP, no TLS, application bytes intact
#
# The identity, audit and policy decision all happened inside
# ztunnel-west. The pod itself has no idea the traffic crossed
# a cluster boundary.

The CRDs behind the flow

Three resources do almost all the work. A Gateway API Gateway declares the east-west listener on :15008. A ServiceEntry tells the source cluster the destination service exists and lives behind the remote gateway IP. A PeerAuthentication in STRICT mode makes the inner mTLS non-optional — STRICT is required; see failure mode four below.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-eastwest
  annotations:
    # Tell the controller to provision a LoadBalancer Service in front
    # of the Envoy Deployment that backs this Gateway.
    networking.istio.io/service-type: LoadBalancer
spec:
  gatewayClassName: istio-eastwest
  listeners:
    - name: cross-network
      port: 15008
      protocol: HBONE
      # The gateway accepts inbound HBONE from any peer cluster that
      # presents a cert in the trust domain below.
      allowedRoutes:
        namespaces:
          from: All

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: checkout-remote
  namespace: payments
spec:
  hosts:
    - checkout.payments.svc.cluster.local
  ports:
    - number: 8080
      name: http
      protocol: HTTP
  location: MESH_INTERNAL    # ← matters: keep this in-mesh, not external
  resolution: DNS
  endpoints:
    # The remote east-west gateway's LoadBalancer IP, *not* the pod IP.
    - address: 203.0.113.42
      network: network-west
      locality: us-east-2/us-east-2a
      ports:
        # ztunnel-east dials 15008 on the gateway, even though the
        # logical service port is 8080.
        http: 15008
      labels:
        cluster: cluster-west

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT   # PERMISSIVE here would silently break identity

What goes wrong (and how it looks)

Five common failure modes for cross-cluster HBONE. Each one breaks at a specific hop above. Match the symptom to the hop and the cause usually follows.

Components on the path

One row per box the packet touches, in order. The Protocol in / Protocol out columns are what each side of that box carries on the wire — use them to map a tcpdump capture or a log line back to the right hop.