Sidecars to Ambient When Your PKI Is RSA

TL;DR

What you'll build

One kind cluster, three app namespaces, one CA. Istio starts in sidecar mode with its built-in CA disabled: every proxy fetches its certificate from istio-csr, which turns each CSR into a cert-manager CertificateRequest, signed by Vault at pki_int/sign/istio-ca.

The PKI mirrors a long-standing RSA estate, all of it in Vault:

In ledger and payments, httpbin serves and a client pod curls the other namespace every two seconds, logging the HTTP code, so each client log is a continuous record of cross-namespace (and later cross-dataplane) health. A fortio pod in ledger provides the bounded zero-downtime scoreboards.

Stand up the RSA baseline

One script builds the world a bank-shaped platform team would recognise: kind, cert-manager, Vault (dev mode), the RSA PKI, the Vault Issuer, istio-csr, and Istio in sidecar mode. The two istiod values that matter are pilot.env.ENABLE_CA_SERVER: "false", which retires istiod as a CA completely, and global.caAddress: cert-manager-istio-csr.cert-manager.svc:443, which points every proxy at istio-csr instead. From that moment Vault is the only thing in the cluster that can sign a workload certificate, and the Vault role is the only place key policy lives.

./scripts/01-setup.sh

vault secrets enable pki
vault write pki/root/generate/internal common_name="Lab Root CA" key_type=rsa key_bits=4096 ttl=87600h

vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal common_name="Lab Intermediate CA" key_type=rsa key_bits=4096
vault write pki/root/sign-intermediate csr=... format=pem_bundle ttl=43800h
vault write pki_int/intermediate/set-signed certificate=...

# THE setting this lab is about — RSA only, like an estate that standardised years ago
vault write pki_int/roles/istio-ca \
  allowed_uri_sans="spiffe://*" allow_any_name=true enforce_hostnames=false require_cn=false \
  server_flag=true client_flag=true \
  key_type=rsa key_bits=2048 \
  ttl=1h max_ttl=24h

helm upgrade -i cert-manager-istio-csr cert-manager-istio-csr \
  --repo https://charts.jetstack.io -n cert-manager --version v0.16.0 -f - <<EOF
app:
  certmanager:
    namespace: istio-system
    preserveCertificateRequests: true
    issuer: { name: istio-ca, kind: Issuer, group: cert-manager.io }
  tls:
    trustDomain: cluster.local
    rootCAFile: /var/run/secrets/istio-csr/ca.pem     # the Vault ROOT — the mesh trust anchor
    certificateDNSNames: [cert-manager-istio-csr.cert-manager.svc]
  server:
    clusterID: Kubernetes
volumeMounts: [{ name: root-ca, mountPath: /var/run/secrets/istio-csr }]
volumes: [{ name: root-ca, secret: { secretName: istio-root-ca } }]
EOF

helm upgrade -i istiod istiod --repo https://istio-release.storage.googleapis.com/charts \
  -n istio-system --version 1.30.3 -f - <<EOF
global:
  caAddress: cert-manager-istio-csr.cert-manager.svc:443
pilot:
  env:
    ENABLE_CA_SERVER: "false"       # istiod no longer signs anything — Vault is the only CA
meshConfig:
  accessLogFile: /dev/stdout
EOF

./scripts/02-deploy-apps.sh
# which runs, visibly:
kubectl apply -f yaml/10-apps/    # namespaces (istio-injection=enabled), apps, PeerAuthentication STRICT

kubectl -n ledger logs deploy/client --tail=3     # ledger -> payments, every 2s
kubectl -n payments logs deploy/client --tail=3   # payments -> ledger

./scripts/03-show-certs.sh      # the whole inventory table — or by hand:

istioctl proxy-config secret deploy/httpbin -n ledger -o json \
  | jq -r '.dynamicActiveSecrets[] | select(.name=="default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -text | grep -E 'Public Key Algorithm|Issuer|Serial' -A1

kubectl -n ledger exec deploy/fortio -c fortio -- \
  fortio load -c 4 -qps 25 -t 10s http://httpbin.payments:8000/status/200 | grep "Code "
# Code 200 : 250 (100.0 %)

Ambient arrives, and nothing moves

Four changes land here, in order, and none of them touches a running application: istio-csr learns to trust ztunnel, istiod flips to the ambient profile, and the CNI agent and ztunnel install. The istio-csr change is the interesting one. A sidecar requests a certificate for its own pod identity, but ztunnel authenticates as itself and requests certificates for the workload identities it fronts on its node. That impersonation model has to be explicitly authorised, and it is why ambient needs istio-csr v0.12.0 or newer: app.server.caTrustedNodeAccounts=istio-system/ztunnel.

kubectl -n ledger exec deploy/fortio -c fortio -- \
  fortio load -c 4 -qps 25 -t 150s http://httpbin.payments:8000/status/200 &

./scripts/04-enable-ambient.sh    # istio-csr caTrustedNodeAccounts + istiod profile=ambient + CNI + ztunnel

helm upgrade cert-manager-istio-csr cert-manager-istio-csr --repo https://charts.jetstack.io \
  -n cert-manager --version v0.16.0 --reuse-values \
  --set "app.server.caTrustedNodeAccounts=istio-system/ztunnel"

helm upgrade istiod istiod --repo https://istio-release.storage.googleapis.com/charts \
  -n istio-system --version 1.30.3 --reuse-values \
  --set profile=ambient --set istio_cni.enabled=true

helm upgrade -i istio-cni cni --repo https://istio-release.storage.googleapis.com/charts \
  -n istio-system --version 1.30.3 --set profile=ambient ...

helm upgrade -i ztunnel ztunnel --repo https://istio-release.storage.googleapis.com/charts \
  -n istio-system --version 1.30.3 \
  --set caAddress=cert-manager-istio-csr.cert-manager.svc:443 ...   # same CA path as everyone else

./scripts/05-interop-roll.sh
# which runs, visibly:
kubectl -n ledger rollout restart deploy/httpbin deploy/client deploy/fortio
kubectl -n ledger rollout status  deploy/httpbin deploy/client deploy/fortio --timeout=180s

# then shows three proofs: istiod's PROTOCOL=HBONE advertisement for every
# ledger workload (istioctl ztunnel-config workloads), the HBONE flag on the
# re-injected pods, and the fresh cert still being RSA:
istioctl ztunnel-config workloads <ztunnel-pod>.istio-system | grep ledger
# ledger  httpbin-...   10.244.1.20  istio-csr-worker  None  HBONE

Break it somewhere safe

The Vault role still says key_type=rsa. Instead of discovering what that means by migrating payments, enrol a namespace that exists to take the hit. preflight is born straight into ambient (the namespace carries istio.io/dataplane-mode=ambient), so the moment its pod starts, ztunnel prefetches a certificate for it, and Vault says no. This is the dev-cluster rehearsal that saves the production estate.

./scripts/06-preflight-break.sh
# enrols the namespace, waits for the rejection, then shows the denied
# CertificateRequest (with Vault's exact error) and ztunnel's log

$ kubectl -n istio-system get certificaterequests
NAME              APPROVED   DENIED   READY   ISSUER     REQUESTER                                         AGE
istio-csr-8k2wp   True                False   istio-ca   system:serviceaccount:cert-manager:cert-manager   41s

$ kubectl -n istio-system get certificaterequest istio-csr-8k2wp \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'
Failed to sign certificate request: Error making API request.
URL: POST http://vault.vault.svc:8200/v1/pki_int/sign/istio-ca
Code: 400. Errors:
* role requires keys of type rsa

The fix: key_type=any, and why not ec

The Vault PKI role API accepts four values for key_type: rsa, ec, ed25519 and any, and documents that key_bits is "ignored … in signing operations when key_type=any". That is precisely the mesh's case: istio-csr and cert-manager only ever use the sign endpoints, where the client generates the key and Vault signs a CSR. (any cannot be used where Vault generates the key itself, the issue endpoints and CA generation, so if other consumers share this role for key-generation, give the mesh its own role. Separate roles per consumer is good Vault hygiene anyway.)

./scripts/07-vault-allow-ec.sh
# which runs, visibly (full parameter set — a role write REPLACES the role):
vault write pki_int/roles/istio-ca \
  allowed_uri_sans="spiffe://*" allow_any_name=true enforce_hostnames=false require_cn=false \
  server_flag=true client_flag=true \
  key_type=any \
  ttl=1h max_ttl=24h

Migrate payments, under load

kubectl -n ledger exec deploy/fortio -c fortio -- \
  fortio load -c 4 -qps 25 -t 90s http://httpbin.payments:8000/status/200 &

./scripts/08-migrate-payments.sh
# which runs, visibly (with the fortio scoreboard managed for you):
kubectl label ns payments istio.io/dataplane-mode=ambient istio-injection- --overwrite
kubectl -n payments rollout restart deploy/httpbin deploy/client
kubectl -n payments rollout status  deploy/httpbin deploy/client --timeout=180s

{"scope":"access","message":"connection complete",
 "src.workload":"client-74888b7c79-6g7mr","src.namespace":"payments",
 "src.identity":"spiffe://cluster.local/ns/payments/sa/client",
 "dst.addr":"10.244.1.29:15008","dst.hbone_addr":"10.244.1.29:8080",
 "dst.service":"httpbin.ledger.svc.cluster.local",
 "dst.identity":"spiffe://cluster.local/ns/ledger/sa/httpbin", ...}

# :15008 + both identities = HBONE mTLS into the sidecar. Before the STEP 5 roll this same
# flow went to :8080 in plaintext with no identities, and STRICT reset it.

NAMESPACE   WORKLOAD   DATAPLANE KEY ALGORITHM    SERIAL                 ISSUER
ledger      httpbin    sidecar   rsaEncryption    24A8082CAD11F94F5660…  CN=Lab Intermediate CA
ledger      client     sidecar   rsaEncryption    7F9127E237F9A9FB7DFD…  CN=Lab Intermediate CA
ledger      fortio     sidecar   rsaEncryption    49B7384281E1831A6D05…  CN=Lab Intermediate CA
payments    httpbin    ambient   id-ecPublicKey   39F5F1CE2894412EF9D2…  CN=Lab Intermediate CA
payments    client     ambient   id-ecPublicKey   02E6DEC7F88A01244E0B…  CN=Lab Intermediate CA
preflight   httpbin    ambient   id-ecPublicKey   27E727F4D5F9FD289E73…  CN=Lab Intermediate CA

Locking the role to EC is a sidecar outage

Set the role to key_type=ec while sidecars still exist and you have started a sidecar outage. New and restarted pods are hit immediately: their RSA CSR bounces and the pod never becomes ready. Every existing sidecar follows within one certificate TTL: its renewal CSR bounces the same way, and the certificate it is serving with expires. With this lab's 1h leaf TTL, the whole estate is dark within the hour.

./scripts/09-sidecar-outage.sh
# role -> ec, scale to 2, show the rejection and the stuck pod,
# role -> any, watch it heal, scale back to 1

Sharing the update with the InfoSec team

In most estates this migration goes through a security change request. Here is what belongs in it: three things to assess, what does not change, and one transitional note.

1. The CA relaxes key-policy enforcement during the migration window. With key_type=rsa the CA refuses anything that is not RSA-2048; with key_type=any it signs whatever key type the CSR carries. The exposure is bounded because only istio-csr can reach this role (through the cert-manager Issuer's Vault policy), and the only CSR generators behind it are istio-agent (RSA-2048) and ztunnel (ECDSA P-256). Keep the role dedicated to the mesh, and once the last sidecar is gone flip it to key_type=ec key_bits=256: the CA becomes a strict control point again, tighter than today. any is the transition posture with a defined exit, not the destination.

2. A new algorithm enters scope: ECDSA P-256. Check it against the approved-algorithms standard, but it is a strength upgrade, not a downgrade: P-256 gives roughly 128-bit security versus roughly 112-bit for RSA-2048, and it is approved in FIPS 186-5 and NCSC guidance. An internal standard that says "RSA-2048 minimum" almost certainly predates EC rather than prohibiting it, and that is exactly what the change request should get confirmed in writing. The chain shape (EC leaf signed by an RSA-4096 intermediate) is standard and valid; the roots, and any HSM backing them, are untouched. If there is a FIPS-validated-module requirement, note that upstream ztunnel builds are not FIPS-validated, while the Solo distribution of Istio ships a FIPS variant.

3. The certificate requester model changes, and this is the biggest item. Today every workload's istio-agent requests a certificate for its own identity, authenticated by its own ServiceAccount token. With ambient, caTrustedNodeAccounts authorises the ztunnel ServiceAccount to request certificates for other workloads' identities (the ones resident on its node). That concentrates trust: compromise of a ztunnel pod means the ability to obtain certificates for any workload on that node. The counter-argument InfoSec will want to hear is that this is node-level blast radius they already carry, since a compromised node can read the SA tokens of every pod on it today, so the effective boundary has not moved. But it is a genuine trust-model change at the CA and should be reviewed and accepted explicitly, not discovered later.

What does not change: the trust anchors (same root, same intermediate, same trust bundle), certificate TTLs, SPIFFE identity naming, SAN policy, STRICT mTLS enforcement, and the Vault audit trail. Every signing operation still lands in Vault's audit log exactly as now, and with preserveCertificateRequests the cluster keeps a durable record of every issuance and rejection, which is arguably an audit improvement. Attribution at the Vault door (requests arriving via the cert-manager Issuer identity rather than per-workload) is already true today with istio-csr, so it is not a delta.

One transitional note: during coexistence the estate serves a mix of RSA and EC leaf certificates. Anything that inspects or pins key types, such as TLS middleboxes or compliance scanners with expected-algorithm rules, should have its expectations updated before migration starts, or it will generate noise that looks like a security event and is not.

Framed that way, the change request is: two config values (key_type=any, caTrustedNodeAccounts), one algorithm addition that raises the security level, one explicitly-accepted trust delegation that stays within existing node blast radius, and a defined stricter end-state. A reviewable, boundable change rather than an open-ended one.

Clean up

./scripts/99-teardown.sh      # kind delete cluster --name istio-csr

See also

• Vault PKI secrets engine API — key_type accepts rsa, ec, ed25519 and any; key_bits is ignored in signing operations when key_type=any.
• cert-manager istio-csr — the agent this lab puts between Istio and Vault; ambient support (trusted CA node accounts) landed in v0.12.0.
• Istio ambient architecture — where ztunnel and HBONE fit.
• Related — Sidecar to Ambient Upgrade: the migration mechanics this lab builds on, with waypoints, canaries and rollback.
• Related — Ambient Deployment and L4/L7 Policies Demo: what the certificates are for — identity-based authorization at L4 and L7.
• Related — Trust and Identity in the Mesh: RootTrustPolicy, cert-manager and Vault as mesh CAs, side by side.

Versions

Built and verified on: