Manual multi-cluster vs Solo Enterprise for Istio management plane

Why this matters

A working multi-cluster mesh is the start of the operator's problem, not the end. Stand-up is a one-time event; Day 2 is forever. The seven production realities below are where the manual path stops scaling — each one is a side-by-side comparison further down the page.

1. 1 Root CA rotation. Rare but high-stakes — driven by compromise, cryptographic obsolescence (e.g. moving from RSA-2048 to RSA-4096), organisational change (M&A, trust-domain restructure), or aged key material. Root certs typically live 10–25 years; the rotation needs to happen without breaking in-flight mTLS across every cluster in the mesh.
2. 2 Intermediate & workload-cert rotation. Intermediates typically rotate every 1–3 years; workload SPIFFE certs every ~24h by default. Every cluster × every pod × every day on the workload tier — at scale, automation isn't optional.
3. 3 Adding a cluster on Day 90. A new region, a new acquisition, a new tenant — without touching the existing N clusters.
4. 4 Audit. "Who allowed cluster X to join the mesh on date Y, with what permissions?" — answered in seconds, not days.
5. 5 RBAC consistency across N clusters. One policy in mgmt → enforced everywhere, no per-cluster drift.
6. 6 Federation across trust domains. M&A — bank acquires fintech, both already on Istio, both with their own root of trust.
7. 7 Observability. A single pane of glass across the mesh — vs N Grafanas, N Kialis, N sets of dashboards that drift.

Side-by-side scenarios

For each scenario, the amber column is what the operator actually types on the manual path; the green column is what the operator types when the Solo Enterprise for Istio management plane is in front of the mesh. Same data plane underneath — different operator experience on top.

# 1. On a trusted ops host: mint a new intermediate from the shared root
./gen-intermediate.sh prod-eu-west-3 \
  --root-ca-cert ~/safe/root-ca.crt \
  --root-ca-key  ~/safe/root-ca.key \
  --out ./prod-eu-west-3-cacerts

# 2. scp the cacerts Secret material onto the new cluster
kubectl --context prod-eu-west-3 create ns istio-system
kubectl --context prod-eu-west-3 -n istio-system create secret generic cacerts \
  --from-file=./prod-eu-west-3-cacerts/

# 3. Stand up Istio Ambient + east-west GW + peering chart on the new cluster
helm install istio-base ... ; helm install istiod-gloo ... ;
helm install istio-cni ... ; helm install ztunnel ... ;
helm install peering ...   # exposes the istio-eastwest GW

# 4. Cross-apply remote-secrets — N*(N-1) operations
for SRC in cluster-1 cluster-2 cluster-3 prod-eu-west-3; do
  for DST in cluster-1 cluster-2 cluster-3 prod-eu-west-3; do
    [ "$SRC" = "$DST" ] && continue
    istioctl create-remote-secret --context $SRC --name $SRC \
      | kubectl --context $DST apply -f -
  done
done

# 5. Restart istiod everywhere so it re-reads the new remote-secrets
for C in cluster-1 cluster-2 cluster-3 prod-eu-west-3; do
  kubectl --context $C -n istio-system rollout restart deploy/istiod-gloo
done

# Single command on the ops host. The mgmt plane already trusts the
# root CA and issues a per-cluster intermediate. The agent installed
# on the new cluster pulls config from the mgmt plane.
meshctl cluster register prod-eu-west-3 \
  --mgmt-context    mgmt \
  --remote-context  prod-eu-west-3 \
  --version         2.12.0

# Verify
kubectl --context mgmt -n gloo-mesh get kubernetescluster prod-eu-west-3
# NAME              AGE   AGENT-VERSION   STATUS
# prod-eu-west-3    34s   2.12.0          Connected

# 1. Generate the new root CA (on a trusted ops host)
./gen-root-ca.sh --out ~/safe/root-ca-2026Q2

# 2. For each cluster: regenerate the intermediate signed by the NEW root,
#    but keep the OLD root in the trust chain during the transition
for C in cluster-1 cluster-2 cluster-3; do
  ./gen-intermediate.sh $C \
    --root-ca-cert ~/safe/root-ca-2026Q2/root-ca.crt \
    --root-ca-key  ~/safe/root-ca-2026Q2/root-ca.key \
    --out ./tmp/$C

  # Patch the cacerts Secret with BOTH roots in ca-cert.pem (old + new)
  cat ~/safe/root-ca-2025Q1/root-ca.crt \
      ~/safe/root-ca-2026Q2/root-ca.crt \
      > ./tmp/$C/ca-cert.pem

  kubectl --context $C -n istio-system create secret generic cacerts \
    --from-file=./tmp/$C/ --dry-run=client -o yaml \
    | kubectl --context $C -n istio-system apply -f -
done

# 3. Rolling istiod restart so it picks up the new intermediate
for C in cluster-1 cluster-2 cluster-3; do
  kubectl --context $C -n istio-system rollout restart deploy/istiod-gloo
  kubectl --context $C -n istio-system rollout status  deploy/istiod-gloo
done

# 4. Wait for every workload pod to have re-issued an SVID from the new int
#    (default rotation period ~24h). Only then is it safe to remove the OLD root.

# 5. Drop the OLD root from ca-cert.pem and re-apply cacerts everywhere

# With Vault as the upstream root of trust:
kubectl --context mgmt -n gloo-mesh apply -f - <<'EOF'
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      generated: {}
    intermediateCertOptions:
      secretRotationGracePeriodRatio: 0.5    # auto re-issue intermediates at 50% TTL
    autoRestartPods: true                    # gracefully roll workloads after rotation
    agentCa:
      vault:                                  # source of truth = Vault
        caPath: pki_root/ca
        csrPath: pki_root/sign/intermediate
        server: https://vault.example.com:8200
        kubernetesAuth:
          role: gloo-mesh-mgmt
EOF

# When the auditor asks "was the rotation successful on cluster-3?"
kubectl --context mgmt -n gloo-mesh get kubernetescluster cluster-3 -o yaml \
  | yq '.status.caStatus'

# For each "source" cluster, rebuild its istio-reader token,
# then apply the resulting kubeconfig Secret on every PEER cluster.
for SRC in cluster-1 cluster-2 cluster-3; do
  # Re-create the bound token (the old one keeps working until TTL expires)
  kubectl --context $SRC -n istio-system create token istio-reader-service-account \
    --duration 8760h > /tmp/$SRC.token

  # Rebuild the kubeconfig Secret YAML with the new token
  ./build-remote-secret.sh \
    --context  $SRC \
    --token    /tmp/$SRC.token \
    --ca-cert  /tmp/$SRC.ca.crt \
    --server   $(kubectl --context $SRC config view -o jsonpath="{.clusters[?(@.name=='$SRC')].cluster.server}") \
    > /tmp/istio-remote-secret-$SRC.yaml

  # Apply it on every PEER cluster
  for DST in cluster-1 cluster-2 cluster-3; do
    [ "$SRC" = "$DST" ] && continue
    kubectl --context $DST apply -f /tmp/istio-remote-secret-$SRC.yaml
  done
done

# For 3 clusters: 6 cross-applies. For 10 clusters: 90.

# Nothing.
#
# The relay agents on workload clusters authenticate to the mgmt
# server via mTLS — using identities the mgmt plane itself issues
# and rotates on the schedule you set in RootTrustPolicy.
#
# For visibility into the rotation cadence:
kubectl --context mgmt -n gloo-mesh get kubernetescluster -o wide
# NAME         STATUS      AGENT VERSION   LAST HEARTBEAT
# cluster-1    Connected   2.12.0          2026-05-15T08:14:33Z
# cluster-2    Connected   2.12.0          2026-05-15T08:14:35Z
# cluster-3    Connected   2.12.0          2026-05-15T08:14:31Z

# overlays/cluster-1/billing-allow-payments.yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: billing-allow-payments
  namespace: billing
spec:
  selector: { matchLabels: { app: billing } }
  action: ALLOW
  rules:
    - from:
      - source:
          principals:
          - "cluster.local/ns/payments/sa/payments"

# ... and the same file copied into overlays/cluster-2/...
# ... and overlays/cluster-3/...
# When the SA name changes, you change it in N places.

# One file — applied to the mgmt cluster.
# The mgmt plane translates this to per-cluster AuthorizationPolicy
# in every workload cluster where billing or payments exists.
apiVersion: security.policy.gloo.solo.io/v2
kind: AccessPolicy
metadata:
  name: billing-allow-payments
  namespace: gloo-mesh
spec:
  applyToDestinations:
    - selector:
        labels: { app: billing }
  config:
    authn:
      mtls: { required: true }
    authz:
      allowedClients:
        - serviceAccountSelector:
            labels: { app: payments }

# Per-cluster: label the Service, then verify
for C in cluster-1 cluster-2 cluster-3; do
  kubectl --context $C -n bookinfo label svc productpage \
    solo.io/service-scope=global --overwrite

  # Did istiod pick it up?
  istioctl --context $C multicluster check | grep "Shared Services"
done

# If one cluster shows zero shared services, you're debugging per-cluster:
#   - is istiod's clusterID right?
#   - is the network label on the namespace?
#   - is the multicluster license actually present (lt: ent, not lt: trial)?

# One file, applied once on mgmt — the Ambient-native global-aliasing CR.
apiVersion: networking.solo.io/v2alpha1
kind: Segment
metadata:
  name: productpage-global
  namespace: gloo-mesh
spec:
  # Workloads anywhere in the mesh dial this hostname; istiod programs the
  # synthetic VIP (240.240.0.x/16) with endpoints from every cluster that
  # has a matching Service.
  hosts:
    - productpage.bookinfo.mesh.internal
  selector:
    namespace: bookinfo
    labels: { app: productpage }
  ports:
    - number: 9080
      protocol: HTTP

# Verify which clusters translated it (which workload clusters now have
# the Service labelled global and istiod programmed the synthetic VIP):
kubectl --context mgmt -n gloo-mesh get segment productpage-global \
  -o jsonpath='{.status.clusters}'

# Best-case reconstruction:
git -C ops/gitops log --diff-filter=A -- "remote-secrets/cluster-prod-eu-west-3*.yaml"
#   commit abc123  Author: Jane Doe  Date: 2026-02-12
#   commit def456  Author: Jane Doe  Date: 2026-02-12

# What trust-domain was used? Inspect the intermediate cert SAN in the
# secret material — assuming it's still in the bucket:
openssl x509 -in ./tmp/prod-eu-west-3/ca-cert.pem -text -noout \
  | grep -A1 "Subject Alternative Name"

# What permissions were granted? The remote-secret is bound to a SA — go
# read that SA's RoleBindings on the source cluster.

# The CR exists from the moment of registration.
kubectl --context mgmt -n gloo-mesh get kubernetescluster -o wide

# NAME              CREATED         AGENT VERSION   TRUST DOMAIN     STATUS
# cluster-1         2025-08-12      2.12.0          cluster.local    Connected
# cluster-2         2025-08-12      2.12.0          cluster.local    Connected
# cluster-3         2025-08-12      2.12.0          cluster.local    Connected
# prod-eu-west-3    2026-02-12      2.12.0          cluster.local    Connected

# Who did it? Kubernetes audit log on the mgmt cluster:
kubectl --context mgmt -n gloo-mesh get events \
  --field-selector involvedObject.name=prod-eu-west-3
# 2026-02-12T10:14Z  Registered  by jane.doe@bank.example.com

# Per-cluster setup that the operator built and now maintains:
#   - Prometheus + node-exporter + kube-state-metrics  ×N clusters
#   - Grafana + dashboards                              ×N clusters
#   - Jaeger / Tempo                                    ×N clusters
#   - (optionally) Thanos / Mimir to federate metrics  — built by you
#
# Investigation flow:
#   1. Open Grafana on cluster-1, find the failing request
#   2. Note the trace ID
#   3. Open Jaeger on cluster-1, find the segment that exits the cluster
#   4. Switch to Jaeger on cluster-2, find the segment that enters
#   5. Repeat for cluster-3
#   6. Reconstruct the timeline in your head

# One UI, fed by the mgmt plane's telemetry pipeline.
# Service topology across the whole mesh, with per-edge latency.
# Cluster-health view: which agents are connected, which aren't.
# Aggregated metrics keyed by service, namespace, AND cluster.

# Same Prometheus data underneath — the difference is that the
# mgmt plane federates it for you.

kubectl --context mgmt -n gloo-mesh port-forward svc/gloo-mesh-ui 8090:8090
# Open http://localhost:8090

Architecture comparison

The visual story behind why the manual path doesn't scale: every edge in the manual diagram is a cross-cluster kubectl apply the operator has to do (and re-do, and audit). In the managed diagram every edge is automated.

            ┌──────────────┐                ┌──────────────┐
            │  cluster-1   │ ◄────────────► │  cluster-2   │
            │  istiod-gloo │   remote-sec   │  istiod-gloo │
            │  east-west   │   peer-bundle  │  east-west   │
            └──────┬───────┘                └──────┬───────┘
                   │                               │
                   │   remote-sec, peer-bundle     │
                   ▼                               ▼
            ┌──────────────────────────────────────────────┐
            │              cluster-3                       │
            │  istiod-gloo, east-west GW, cacerts Secret   │
            └──────────────────────────────────────────────┘

  Each ◄────► edge is TWO kubectl applies + the cert material to back them.
  For N clusters the operator owns N*(N-1) such edges.
  Adding cluster-4: 6 NEW edges (2 per existing peer).
  Rotating root CA: every cacerts Secret on every cluster, in lockstep.

       ┌────────────────────────────────────────────────────┐
       │   Solo Enterprise for Istio management plane       │
       │                                                    │
       │   KubernetesCluster CRs                            │
       │   RootTrustPolicy                                  │
       │   AccessPolicy / Workspace                         │
       │   Segment  (Ambient federation)                    │
       │   Gloo UI  (federated metrics)                     │
       └──────┬───────────────┬───────────────┬─────────────┘
              │               │               │
              ▼               ▼               ▼
       ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
       │ cluster-1   │  │ cluster-2   │  │ cluster-3   │
       │ gloo-mesh-  │  │ gloo-mesh-  │  │ gloo-mesh-  │
       │ agent       │  │ agent       │  │ agent       │
       │ istiod-gloo │  │ istiod-gloo │  │ istiod-gloo │
       │ east-west   │  │ east-west   │  │ east-west   │
       └─────────────┘  └─────────────┘  └─────────────┘

  Each line is one mTLS pipe the mgmt plane owns end-to-end.
  Adding cluster-4: ONE new edge (mgmt → cluster-4).
  Rotating root CA: ONE RootTrustPolicy change, mgmt plane rolls it.
  Data-plane (cluster-to-cluster HBONE) still flows direct — same data plane,
  the mgmt plane is for trust + policy + audit, not for proxying traffic.

When to use which

The manual path is genuinely fine for some real use-cases — the management plane isn't always the right answer. Below is the decision matrix: pick the column whose row matches your situation.

How you'd actually install it

This section walks through the install as it ships on Solo Enterprise for Istio 2.12.4. The shape is the same in production (on cloud Kubernetes or on-prem); local-dev with kind has one extra step around east-west GW exposure, called out explicitly below.

# 1. CRDs on EVERY cluster (mgmt + workload). installEnterpriseCrds=true is
#    what brings in AccessPolicy / Workspace / WorkspaceSettings — the CRs the
#    mgmt plane uses to translate central policy into per-cluster
#    AuthorizationPolicy. The Solo documented Ambient pattern uses
#    installEnterpriseCrds=false; do NOT use that pattern if you want
#    centralised RBAC — flip to true and resolve any agentgateway-crds
#    co-install conflict (the two charts collide on authconfigs +
#    ratelimitconfigs; uninstall enterprise-agentgateway on the mgmt cluster
#    or keep agentgateway off the mgmt-plane cluster entirely).
helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
  --kube-context $MGMT -n gloo-mesh --create-namespace \
  --version 2.12.4 \
  --set installEnterpriseCrds=true \
  --set featureGates.ConfigDistribution=true     # mgmt only

helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
  --kube-context $WORKLOAD -n gloo-mesh --create-namespace \
  --version 2.12.4 \
  --set installEnterpriseCrds=true

# 2. Management plane (mgmt-server + UI + Prometheus + relay + cert-gen job)
#    on the mgmt cluster. The chart generates the relay-*-tls-secret family
#    on first install; never partial-rollout these (an out-of-sync chain
#    across relay-root / relay-server / relay-client breaks the agent
#    handshake — clean uninstall + reinstall is the production-safe recovery).
helm upgrade -i gloo-platform gloo-platform/gloo-platform \
  --kube-context $MGMT -n gloo-mesh \
  --version 2.12.4 -f mgmt-values.yaml \
  --set licensing.glooMeshLicenseKey=$GLOO_MESH_LICENSE_KEY

# 3. Each workload cluster: cross-apply the relay TLS secrets from the mgmt
#    cluster, then install the agent. The agent dials the mgmt-server over
#    mTLS using those secrets — its serverAddress points at the mgmt-server
#    Service IP (LoadBalancer recommended for cross-host reach).
kubectl --context $MGMT -n gloo-mesh get secret relay-root-tls-secret \
  -o yaml | kubectl --context $WORKLOAD -n gloo-mesh apply -f -
# (repeat for relay-client-tls-secret + relay-identity-token-secret)

helm upgrade -i gloo-platform gloo-platform/gloo-platform \
  --kube-context $WORKLOAD -n gloo-mesh \
  --version 2.12.4 -f agent-values.yaml \
  --set licensing.glooMeshLicenseKey=$GLOO_MESH_LICENSE_KEY

# 4. Register the workload cluster on the mgmt plane (one CR).
kubectl --context $MGMT apply -f - <<'EOF'
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata: { name: prod-eu-west-3, namespace: gloo-mesh }
spec:    { clusterDomain: cluster.local }
EOF

# 5. Verify on the mgmt cluster:
kubectl --context $MGMT -n gloo-mesh get kubernetescluster
# NAME              STATUS
# mgmt              ACCEPTED
# prod-eu-west-3    ACCEPTED

Cross-cluster peering on Ambient: auto vs manual

Once both clusters are registered with the mgmt plane, they need data-plane peering — the istio-remote Gateway in each cluster that points at the other's east-west GW so HBONE traffic can flow. In Solo Enterprise for Istio 2.12 there are two paths to wire this. Both ride on the same shared root CA + per-cluster intermediates (in the cacerts Secret) that the manual lab on the left already establishes — Ambient's trust pipeline is bootstrapped outside the mgmt plane, which is also why the older RootTrustPolicy CR is documented as superseded in the Ambient install guide.

helm upgrade gloo-platform gloo-platform/gloo-platform \
  --kube-context $MGMT -n gloo-mesh \
  --set featureGates.ConfigDistribution=true

kubectl --context $C -n istio-system patch deploy istiod-gloo --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-",
        "value":{"name":"PEERING_AUTOMATIC_LOCAL_GATEWAY","value":"true"}}]'

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: manual-peer-west-mini
  namespace: istio-eastwest
  annotations:
    gateway.istio.io/service-account: istio-eastwest   # MUST match the peer's
    gateway.istio.io/trust-domain: cluster.local       # east-west GW SA — SPIFFE
  labels:                                              # SAN verification uses this
    topology.istio.io/cluster: west-mini
    topology.istio.io/network: west-mini
spec:
  gatewayClassName: istio-remote
  addresses:
    - { type: IPAddress, value: 192.168.1.18 }         # peer's east-west GW
  listeners:                                            # LAN-routable IP
    - { name: cross-network, port: 15008, protocol: HBONE, tls: { mode: Passthrough }, allowedRoutes: { namespaces: { from: Same } } }
    - { name: xds-tls,       port: 15012, protocol: TLS,   tls: { mode: Passthrough }, allowedRoutes: { namespaces: { from: Same } } }

./scripts/super-quick.sh                       # prompts for SSH user / host
./scripts/super-quick.sh --user <user> --host <host>   # non-interactive