Gloo Operator — deploy Istio Ambient across N clusters from one YAML

Single-cluster Istio Ambient is a five-minute install. Helm chart, istioctl install, done. That breaks at two clusters — they need to share an identity domain, keep their Istio versions in lock-step, and stand up east-west gateways with the same config. A for loop of helm install with per-cluster values files is not how you want to manage that, and it gets worse with every cluster you add.

The mental model: in the mgmt cluster you write a handful of CRs that describe what the fleet should look like. The Gloo Operator does the rest — installs istiod and ztunnel everywhere, rolls out the east-west gateways, distributes the CA, applies tenancy. You don't ssh into a workload cluster. You don't run helm against it. Drift gets corrected on the next reconcile. The rest of this page is the YAML you write — in one place — to get there.

One management cluster, three workload clusters

lifecycle CRDs multicluster CRDs trust & identity CRDs tenancy CRDs operator / control plane

How to read this: the four CRD columns are the whole API surface — and they all live in the management cluster. You never write YAML anywhere else. gloo-operator watches those CRs and, for each KubernetesCluster that's registered, pushes the right manifests, drops the Secrets and CA bundles in place, and serves runtime config back to a per-cluster gloo-mesh-agent. The output: three workload clusters with identical Istio Ambient builds and a shared trust domain. No drift, no per-cluster snowflakes.

What the operator actually does, on a loop

Nothing magical. Every controller-runtime tick it re-reads each CRD, diffs against what's actually running in each workload cluster, and closes the gap. That's the bit that lets you walk away.

Watch CRDsin mgmt cluster

→

Diff vs cluster stateper workload cluster

→

Apply manifestsistiod · gateways · ztunnel

→

Update statusphase, conditions

Each lifecycle CRD has a .status.clusters[] — one entry per workload cluster. kubectl get istiolifecyclemanager -o yaml from the mgmt cluster shows whether cluster-east, cluster-west and cluster-central are on the desired Istio version, or which one is still rolling. That's the "single pane of glass" — except it's just kubectl.

The CRDs, group by group

🧭 Multicluster multicluster.solo.io/v1alpha1

Step one — and the only step that's even slightly fiddly — is telling the operator which clusters exist and how to reach them. One KubernetesCluster CR per workload cluster, plus a bootstrap Secret with its kubeconfig. After this, nobody touches the workload clusters by hand again.

KubernetesCluster · register cluster-eastapplied in the management cluster

apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata:
  name: cluster-east
  namespace: gloo-mesh
spec:
  clusterDomain: cluster.local
  # The operator drops a kubeconfig here when the workload cluster
  # is onboarded with `meshctl cluster register`.
  kubeConfig:
    secretName: cluster-east-kubeconfig
    secretNamespace: gloo-mesh

KubernetesCluster ×3 · all three workload clusterseast / west / central

---
apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata: { name: cluster-east,    namespace: gloo-mesh }
spec:   { clusterDomain: cluster.local }
---
apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata: { name: cluster-west,    namespace: gloo-mesh }
spec:   { clusterDomain: cluster.local }
---
apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata: { name: cluster-central, namespace: gloo-mesh }
spec:   { clusterDomain: cluster.local }

⚙️ Lifecycle admin.gloo.solo.io/v2

These are the CRs you write the most. Each one says "this is what should be installed on these clusters" — the operator turns it into a per-cluster Helm release and rolls it out. Same chart, same version, same values, n clusters. This is where the operator earns its keep.

IstioLifecycleManager · install Istio Ambient on all 3 clustersone CRD ⇒ three rollouts

apiVersion: admin.gloo.solo.io/v2
kind: IstioLifecycleManager
metadata:
  name: ambient-fleet
  namespace: gloo-mesh
spec:
  installations:
    - clusters:
        - name: cluster-east
          defaultRevision: true
        - name: cluster-west
          defaultRevision: true
        - name: cluster-central
          defaultRevision: true
      revision: 1-26
      istioOperatorSpec:
        profile: ambient
        components:
          cni:
            enabled: true
          ztunnel:
            enabled: true
        values:
          global:
            meshID: solo-mesh
            multiCluster: { clusterName: "" }   # filled per cluster
            network: ""                          # filled per cluster

GatewayLifecycleManager · roll out east-west gatewaysopens HBONE :15008 in every cluster

apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
  name: eastwest-fleet
  namespace: gloo-mesh
spec:
  installations:
    - clusters:
        - { name: cluster-east }
        - { name: cluster-west }
        - { name: cluster-central }
      gatewayRevision: 1-26
      istioOperatorSpec:
        profile: empty
        components:
          ingressGateways:
            - name: istio-eastwestgateway
              namespace: istio-eastwest
              enabled: true
              k8s:
                service:
                  type: LoadBalancer
                  ports:
                    - port: 15008
                      targetPort: 15008
                      name: tls
                      protocol: TCP

CNILifecycleManager · ship istio-cni-node to every clusterprereq for ambient redirection

apiVersion: admin.gloo.solo.io/v2
kind: CNILifecycleManager
metadata:
  name: cni-fleet
  namespace: gloo-mesh
spec:
  installations:
    - clusters:
        - { name: cluster-east }
        - { name: cluster-west }
        - { name: cluster-central }
      revision: 1-26
      istioOperatorSpec:
        profile: ambient
        components:
          cni:
            enabled: true
            namespace: istio-system

🔐 Trust & Identity admin.gloo.solo.io/v2

For cross-cluster mTLS to work, every istiod has to sign workload certs with intermediates that chain back to the same root. Doing that by hand — copying intermediate keys between clusters — is the part of multicluster Istio that breaks under any kind of churn. The operator does it. You point RootTrustPolicy at cert-manager or Vault and the CA bundle lands in every cluster.

RootTrustPolicy · cert-manager as the shared rootone trust domain across 3 clusters

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: shared-trust
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      generated: {}
    intermediateCertOptions:
      secretRotationGracePeriodRatio: 0.10
    autoRestartPods: true
    agentCa:
      certManager:
        issuerRef:
          group: cert-manager.io
          kind: ClusterIssuer
          name: gloo-mesh-root

RootTrustPolicy · existing Vault PKIenterprise CA story

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: shared-trust
  namespace: gloo-mesh
spec:
  config:
    agentCa:
      vault:
        caPath: pki_int/sign/gloo-mesh
        rolePath: pki_int/roles/gloo-mesh
        server: https://vault.internal:8200
        authMethod:
          kubernetes:
            mountPath: kubernetes
            role: gloo-mesh

🗂️ Tenancy admin.gloo.solo.io/v2

With Ambient healthy across all three clusters, you've got a fabric. What you don't have is opinions about who's allowed to call what. That's Workspace — a logical boundary owned by a team or app, with explicit import / export rules. The first Workspace per environment is the line between a sandbox and a product.

Workspace · a payments team spanning all 3 clustersselects ns + clusters

apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: payments
  namespace: gloo-mesh
spec:
  workloadClusters:
    - name: cluster-east
      namespaces:
        - name: payments-prod
    - name: cluster-west
      namespaces:
        - name: payments-prod
    - name: cluster-central
      namespaces:
        - name: payments-prod

WorkspaceSettings · let payments import from "platform"explicit cross-tenant policy

apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
  name: payments
  namespace: payments-prod
spec:
  importFrom:
    - workspaces:
        - name: platform
      resources:
        - kind: SERVICE
          labels:
            shared: "true"
  exportTo:
    - workspaces:
        - name: payments
      resources:
        - kind: SERVICE

VirtualDestination · a global "checkout" hostnameresolves to any healthy cluster

apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: checkout
  namespace: payments-prod
spec:
  hosts:
    - checkout.payments.global
  ports:
    - number: 8080
      protocol: HTTP
  services:
    - labels: { app: checkout }

CLI equivalents — `meshctl`, `helm`, `kubectl`

Every CRD above also has an imperative way in. Use these for bootstrapping a fresh demo, scripting CI, or poking at a cluster that's misbehaving when you need an answer faster than a reconcile cycle. The four blocks below line up with the four CRD groups in the diagram.

🚀 Bootstrap the management plane prerequisite

Drops gloo-operator, gloo-mesh-mgmt-server and the Gloo CRDs into the management cluster. You run this once in the lifetime of the platform — every other CR on this page assumes it's already there.

meshctl install · management profilesimplest path

# Pre-flight — make sure both the kubecontext and the licence are set
export MGMT=gloo-mgmt
export GLOO_MESH_LICENSE_KEY=...

meshctl install \
  --profile mgmt-server \
  --kubecontext=$MGMT \
  --version 2.7.0

helm install · same thing, GitOps-friendlyCRDs + chart

export MGMT=gloo-mgmt
helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

# 1. CRDs first
helm upgrade --install gloo-platform-crds gloo-platform/gloo-platform-crds \
  --kube-context=$MGMT \
  --namespace gloo-mesh --create-namespace \
  --version 2.7.0

# 2. Management plane (gloo-operator + mgmt-server)
helm upgrade --install gloo-platform gloo-platform/gloo-platform \
  --kube-context=$MGMT \
  --namespace gloo-mesh \
  --version 2.7.0 \
  --values mgmt-values.yaml

🧭 Register the workload clusters multicluster

Same outcome as hand-writing three KubernetesCluster CRs and three Secrets — but in one command per cluster. meshctl cluster register mints the bootstrap kubeconfig Secret, creates the CR, and (with --install-agent) drops gloo-mesh-agent on the workload cluster. This is the one place the imperative path beats the declarative one.

meshctl cluster register · one command per clusterx3 for east / west / central

export MGMT=gloo-mgmt

for C in cluster-east cluster-west cluster-central; do
  meshctl cluster register $C \
    --kubecontext=$MGMT \
    --remote-context=$C \
    --version 2.7.0 \
    --install-agent           # also rolls gloo-mesh-agent into $C
done

# Confirm the three KubernetesCluster CRs landed in the mgmt cluster
kubectl --context=$MGMT -n gloo-mesh get kubernetescluster

meshctl cluster deregister · tear one back downcleans agent + CR + Secret

meshctl cluster deregister cluster-east \
  --kubecontext=gloo-mgmt \
  --remote-context=cluster-east \
  --uninstall-agent

⚙️ Install Istio Ambient + east-west gateways lifecycle

Operator path: apply one IstioLifecycleManager and walk away. Imperative path: per-cluster Helm install of Solo's Istio distribution, then a per-cluster east-west gateway install. Use the operator path in production; the helm path is useful for understanding exactly what gets installed.

meshctl install · ambient profile on every clusteristiod + ztunnel + CNI

for C in cluster-east cluster-west cluster-central; do
  meshctl install \
    --profile ambient \
    --kubecontext=$C \
    --version 2.7.0
done

helm install · Solo's Istio Ambient chartsCNI → ztunnel → istiod → gateways

REV=1-26
ISTIO_REPO=https://storage.googleapis.com/istio-enterprise/helm-charts

for C in cluster-east cluster-west cluster-central; do
  helm repo add istio-enterprise $ISTIO_REPO

  # 1. istio-cni
  helm upgrade --install istio-cni istio-enterprise/cni \
    --kube-context=$C --namespace kube-system \
    --version $REV --values cni.yaml

  # 2. ztunnel (DaemonSet, per node)
  helm upgrade --install ztunnel istio-enterprise/ztunnel \
    --kube-context=$C --namespace istio-system --create-namespace \
    --version $REV --values ztunnel.yaml

  # 3. istiod-gloo (Solo's control plane)
  helm upgrade --install istiod istio-enterprise/istiod \
    --kube-context=$C --namespace istio-system \
    --version $REV --values istiod.yaml

  # 4. east-west gateway (HBONE :15008)
  helm upgrade --install istio-eastwestgateway istio-enterprise/gateway \
    --kube-context=$C --namespace istio-eastwest --create-namespace \
    --version $REV --values eastwest.yaml
done

kubectl rollout status · watch the agent reconciledeclarative path only

# When you applied IstioLifecycleManager instead, the operator pushes
# manifests to each cluster. Watch them land:
for C in cluster-east cluster-west cluster-central; do
  echo "--- $C ---"
  kubectl --context=$C -n istio-system rollout status deploy/istiod
  kubectl --context=$C -n istio-system rollout status ds/ztunnel
  kubectl --context=$C -n istio-eastwest rollout status deploy/istio-eastwestgateway
done

# And read .status.clusters[] back in the mgmt cluster
kubectl --context=gloo-mgmt -n gloo-mesh \
  get istiolifecyclemanager ambient-fleet -o yaml

🔍 Day-2 — check, describe, debug ops

Fleet's up. Now what? Most post-install work is meshctl check, meshctl describe, and kubectl get against the lifecycle CRDs to read .status.clusters[]. The commands below are the ones worth keeping in shell history.

meshctl check · health of one clusterconnectivity + agent + CA

# Run from anywhere with both contexts available
meshctl check \
  --kubecontext=cluster-east

# Just the agent ↔ mgmt-server relay channel
meshctl check relay \
  --kubecontext=gloo-mgmt \
  --remote-context=cluster-east

meshctl describe · explain a Workspace or VirtualDestinationresolves imports / exports

# Why can payments call platform.checkout? meshctl walks the
# Workspace + WorkspaceSettings graph and prints the resolved policy.
meshctl describe workspace payments \
  --kubecontext=gloo-mgmt

meshctl describe virtualdestination checkout \
  --namespace payments-prod \
  --kubecontext=gloo-mgmt

kubectl · raw inspection of lifecycle CRDsalways available

MGMT=gloo-mgmt

# Which clusters are registered?
kubectl --context=$MGMT -n gloo-mesh get kubernetescluster

# How is each cluster doing against the desired Istio version?
kubectl --context=$MGMT -n gloo-mesh \
  get istiolifecyclemanager ambient-fleet \
  -o jsonpath='{.status.clusters}' | jq

# Same for gateways + trust
kubectl --context=$MGMT -n gloo-mesh get gatewaylifecyclemanager
kubectl --context=$MGMT -n gloo-mesh get roottrustpolicy

meshctl uninstall · tear everything downdestructive

# Workload clusters first (agents + Istio)
for C in cluster-east cluster-west cluster-central; do
  meshctl uninstall --kubecontext=$C
done

# Then the management plane
meshctl uninstall --kubecontext=gloo-mgmt

Full reference table

Every CRD the operator watches, where it's written (always the mgmt cluster — that's the whole point) and what it ends up producing in the workload clusters.

Resource	Group	API	What it does	Produces in workload cluster
`KubernetesCluster`	multicluster	`multicluster.solo.io/v1alpha1`	Registers a workload cluster with the operator. References a Secret holding its kubeconfig.	none — bootstrap only
`ClusterDomain`	multicluster	`multicluster.solo.io/v1alpha1`	Declares the per-cluster DNS suffix (default `cluster.local`) for service discovery.	istiod-gloo config
`IstioLifecycleManager`	lifecycle	`admin.gloo.solo.io/v2`	Declares the desired Istio Ambient version, profile and overrides; rolls it out per cluster.	istiod-gloo Deployment
`ZTunnelLifecycleManager`	lifecycle	`admin.gloo.solo.io/v2`	Pins the ztunnel image / flags per cluster — usually slaved to the same revision as istiod.	ztunnel DaemonSet
`CNILifecycleManager`	lifecycle	`admin.gloo.solo.io/v2`	Installs `istio-cni-node` so pod traffic is redirected into ztunnel.	istio-cni-node DaemonSet
`GatewayLifecycleManager`	lifecycle	`admin.gloo.solo.io/v2`	Rolls out east-west gateways (HBONE :15008) and / or north-south ingress gateways.	istio-eastwestgateway · ingress envoy
`RootTrustPolicy`	trust	`admin.gloo.solo.io/v2`	Defines the shared root + intermediate CA story (generated / cert-manager / Vault).	cacerts Secret in istio-system
`CASource`	trust	`admin.gloo.solo.io/v2`	Names the issuer (e.g. cert-manager ClusterIssuer) that `RootTrustPolicy` uses.	cacerts Secret
`IdentityProvider`	trust	`admin.gloo.solo.io/v2`	SPIFFE trust-domain configuration shared across all clusters.	istiod-gloo mesh config
`Workspace`	tenancy	`admin.gloo.solo.io/v2`	Logical boundary that owns a set of namespaces across one or more clusters.	RBAC + Istio AuthorizationPolicy
`WorkspaceSettings`	tenancy	`admin.gloo.solo.io/v2`	Declares what a Workspace can import from / export to other Workspaces.	ServiceEntry · AuthorizationPolicy
`VirtualDestination`	tenancy	`networking.gloo.solo.io/v2`	A global hostname that resolves to healthy backends in any cluster.	ServiceEntry + endpoints
`gloo-operator` Deployment	controller	—	The controller-runtime pod that reconciles every CRD above.	runs in mgmt cluster
`gloo-mesh-mgmt-server`	controller	—	xDS aggregator + SPIFFE CSR endpoint — every workload cluster's `gloo-mesh-agent` connects here.	runs in mgmt cluster
`gloo-mesh-agent`	controller	—	Per-workload-cluster agent. Receives config + identities from the mgmt server and applies them locally.	DaemonSet / Deployment in each workload cluster

Where to go from here

Read this alongside the Istio Ambient CRDs visual map. The split: the operator's lifecycle CRDs decide what gets installed; the per-cluster Istio CRDs decide how that installation behaves. With both in hand, adding a fourth cluster is two lines — one new KubernetesCluster and one extra name on the existing IstioLifecycleManager. That's the whole pitch.

The exhaustive reference is in the Solo docs. For an end-to-end run on a laptop, the Istio multicluster on kind walk-through is the place to start.