Trust & identity — from RootTrustPolicy to an mTLS handshake

🔍 Day-2 — verify trust is wired end-to-end ops

Walk the chain the same way the bytes flow. Relay healthy? Did cacerts show up in istio-system? Is the pod (or ztunnel) actually serving the right SDS? And finally, what does the gateway hand out when something on the other side of the internet asks? If all four answer yes, trust is wired.

# The intermediate + signing key are pushed from the mgmt-server to
# each cluster's gloo-mesh-agent over the relay channel (gRPC mTLS).
# If relay is unhealthy, cert rotation will silently lag — so this
# is the first thing to check when SVIDs go stale.
meshctl check relay \
  --kubecontext=gloo-mgmt \
  --remote-context=cluster-east

# Full health check (relay + agent + CA reconcile state):
meshctl check --kubecontext=cluster-east

# The agent writes the intermediate + key into a Secret called
# `cacerts` in istio-system. istiod-gloo mounts that Secret and
# uses it as its SPIFFE CA.
for C in cluster-east cluster-west cluster-central; do
  echo "--- $C ---"
  kubectl --context=$C -n istio-system get secret cacerts \
    -o jsonpath='{.data.ca-cert\.pem}' \
    | base64 -d \
    | openssl x509 -noout -subject -issuer -dates
done

# Sanity-check: every cluster's intermediate should share the same
# Issuer DN (= the shared root).

# A sidecar holds two SDS secrets:
#   default  — the workload's leaf SVID (rotated automatically)
#   ROOTCA   — the trust bundle it validates peers against
istioctl proxy-config secret -n payments deploy/checkout

# Expected output: 2 dynamic active secrets, both Valid. If "default"
# is in state Warming or Stale, the SDS dance with istiod is failing.

# Ambient mode: each pod's identity is held by the local ztunnel,
# not by a sidecar. Hit ztunnel's admin endpoint to see the SVIDs
# it currently has and which workloads they map to.
kubectl -n istio-system exec ds/ztunnel -- \
  curl -s http://localhost:15000/config_dump \
  | jq '.workloads, .certificates'

# Newer ztunnel revisions expose a dedicated CLI:
#   kubectl -n istio-system exec ds/ztunnel -- ztunnel admin certs
# Check `ztunnel --help` on the binary in your cluster — the exact
# subcommand name has moved between revisions.

# Confirm two clusters' east-west gateways chain to the same root.
for CTX in cluster-east cluster-west cluster-central; do
  IP=$(kubectl --context=$CTX -n istio-eastwest \
    get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo "--- $CTX  ($IP) ---"
  openssl s_client \
    -connect $IP:15008 \
    -servername outbound_.15008_._.eastwest-gw.istio-eastwest.svc.cluster.local \
    -showcerts < /dev/null 2>/dev/null \
    | openssl x509 -noout -issuer
done

# Every line should show the same Issuer DN — that's the root from
# RootTrustPolicy.

Area 1 · mTLS client auth Rogue workload with no SVID

Launch a curl pod in a namespace that is not ambient enrolled — it has no SPIFFE identity and no ztunnel on the wire to initiate HBONE. Then try to reach a service behind a STRICT-mTLS waypoint.

$ kubectl --context=cluster-east create ns rogue
$ kubectl --context=cluster-east -n rogue run rogue \
    --image=curlimages/curl -i --tty --rm -- \
    curl -is http://reviews.payments.svc:9080 | head -3
curl: (52) Empty reply from server

Config snippet — the PeerAuthentication that makes plaintext a non-starter:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: payments }
spec: { mtls: { mode: STRICT } }

Ztunnel log on the receiving node — connection dropped before any L7 byte:

$ kubectl --context=cluster-east -n istio-system logs ds/ztunnel \
    --tail=2 | jq -c .
{"level":"info","msg":"connection refused: peer is not part of the mesh",
 "src.ip":"10.244.3.17","dst.workload":"reviews",
 "dst.service":"payments/reviews","reason":"no_hbone_no_identity"}

Area 1 · SPIFFE-keyed AuthZ Workload from a denied namespace

Both source and destination are ambient (both have SVIDs) but an AuthorizationPolicy on the destination filters on source.principals. Place a curl pod in marketing instead of payments and call reviews.

$ kubectl --context=cluster-east -n marketing exec deploy/sleep -- \
    curl -is http://reviews.payments.svc:9080/healthz | head -3
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain

RBAC: access denied

Config snippet — only the checkout SA in payments can reach reviews; anything else, including a peer in the same mesh, is rejected by SPIFFE URI:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: reviews-only-checkout, namespace: payments }
spec:
  selector: { matchLabels: { app: reviews } }
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/payments/sa/checkout"]

Waypoint access log — the SPIFFE URI of the caller is the only thing that drove the decision, no header spoof would change it:

$ kubectl --context=cluster-east -n payments logs deploy/payments-waypoint \
    --tail=1 | jq -c '{src:.source.principal, decision:.rbac.decision, policy:.rbac.matched}'
{"src":"spiffe://cluster.local/ns/marketing/sa/sleep",
 "decision":"deny","policy":"reviews-only-checkout"}

Area 1 · Trust domain SVID signed by a different root

Pretend a fourth cluster (cluster-rogue) is enrolled with its own istiod and its own self-signed root — i.e. the operator never ran RootTrustPolicy for it. Its ztunnel tries to open an HBONE tunnel to cluster-east's east-west gateway. The handshake fails before any payload byte:

# From cluster-rogue, attempting to reach cluster-east:
$ EAST_IP=$(kubectl --context=cluster-east -n istio-eastwest \
    get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ openssl s_client -connect $EAST_IP:15008 \
    -cert /etc/certs/rogue-svid.pem -key /etc/certs/rogue-key.pem \
    -CAfile /etc/certs/rogue-root.pem 2>&1 | head -8
CONNECTED(00000003)
depth=2 O = istio.example, CN = Rogue Root CA
verify error:num=19:self-signed certificate in certificate chain
verify return:0
SSL_connect:error in SSLv3/TLS read server certificate verify (alert 48 — unknown_ca)

Config snippet — the trust bundle that cluster-east's ztunnel uses to validate peers is built from the shared root pinned by RootTrustPolicy. Any cert chain that doesn't terminate in that bundle is unknown_ca:

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata: { name: root-trust-policy, namespace: gloo-mesh }
spec:
  config:
    mgmtServerCa:
      generated: {}                     # one root, pinned cluster-wide
    autoRestartPods: true
    intermediateCertOptions:
      ttlDays: 365

Ztunnel log on the receiver side — note no src.workload, because the handshake never reached the HBONE inner header:

$ kubectl --context=cluster-east -n istio-system logs ds/ztunnel \
    --tail=1 | jq -c .
{"level":"warn","msg":"tls handshake failed",
 "src.ip":"203.0.113.14","reason":"x509: certificate signed by unknown authority",
 "phase":"hbone_inbound"}

Area 4 · Outbound mTLS Upstream presents an unverified cert

Now flip direction. The waypoint in payments is configured to mTLS to an external upstream (an enterprise API outside the mesh) using a specific CA bundle. The upstream rotates its leaf to one signed by a CA not in that bundle. The egress call fails closed:

$ kubectl --context=cluster-east -n payments exec deploy/checkout -- \
    curl -is https://billing.partner.example.com/charge | head -3
HTTP/1.1 503 Service Unavailable
x-envoy-upstream-service-time: 7
content-length: 95

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Config snippet — the DestinationRule pins the upstream's CA. Cert rotation outside that bundle = fail-closed, no fallback to system roots:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: billing-partner, namespace: payments }
spec:
  host: billing.partner.example.com
  trafficPolicy:
    tls:
      mode: MUTUAL
      clientCertificate: /etc/certs/partner-client.pem
      privateKey:        /etc/certs/partner-client-key.pem
      caCertificates:    /etc/certs/partner-ca-bundle.pem
      sni:               billing.partner.example.com

Waypoint log — the response_code_details name the exact handshake failure, distinct from a TCP reset:

$ kubectl --context=cluster-east -n payments logs deploy/payments-waypoint \
    --tail=1 | jq -c '{up:.upstream_host, code:.response_code, details:.response_code_details}'
{"up":"billing.partner.example.com:443","code":503,
 "details":"upstream_reset_before_response_started{remote_reset|TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED}"}