Solo Enterprise AgentGateway & Ambient Multicluster Mesh on Kind

Clone this repo and set environment variables

Every command below runs from the repo root — clone first, then export the env vars in every terminal tab you use:

git clone git@github.com:tjorourke/solo.git
cd solo/agentgw-multi-cluster-kind

# Cluster names — suffixed -ag so this demo can run alongside the istio-gw demo
export CLUSTER1=kind-east-ag
export CLUSTER2=kind-west-ag

# Solo enterprise licences (required) — request a trial from
# https://www.solo.io/free-trial/ if you don't have keys.
export SOLO_ISTIO_LICENSE_KEY="eyJ..."
export AGENTGATEWAY_LICENSE_KEY="eyJ..."

# Optional — only needed if you run STEP 08 (Gloo UI management plane).
export GLOO_MESH_LICENSE_KEY="eyJ..."

# Or point at a sourceable file (any shell script that exports both keys)
# SECRETS_FILE=/path/to/secrets.sh   # used by ./scripts/quick.sh

# Version pins
export ISTIO_VERSION=1.29.2-solo
export ISTIO_VERSION_OPERATOR="${ISTIO_VERSION%-solo}"   # strips "-solo" for the operator
export GLOO_OPERATOR_VERSION=0.5.2
export GATEWAY_API_VERSION=v1.4.0
export AGW_VERSION=v2.3.3  # chart tag — note the 'v' prefix (chart at v2.2+ uses it; appVersion/image tag is plain 2.3.3)

Create east-ag + west-ag kind clusters

The script reads $CLUSTER1 and $CLUSTER2, strips the kind- prefix to get the cluster name, and looks up the matching config in kind/. The -ag suffix means this demo can run alongside the istio-gw demo without name collisions.

./scripts/01-clusters.sh

Expected output:

  ✓ [east-ag] ready
  ✓ [west-ag] ready

Contexts available:
*  kind-east-ag   ← $CLUSTER1
   kind-west-ag   ← $CLUSTER2

Install MetalLB — LoadBalancer IPs on kind

Replaces the cloud LB. Detects the Docker kind network IPv4 CIDR at runtime and assigns non-overlapping pools. Pools are .100–.110 (east-ag) and .120–.130 (west-ag) — no overlap with the istio-gw demo's .200–.230 range.

./scripts/02-metallb.sh

Expected output:

  docker kind network: 172.22.0.0/16  (base: 172.22)
  ✓ [east-ag] MetalLB controller ready
  ✓ [west-ag] MetalLB controller ready
  ✓ MetalLB pools configured  (east-ag .100-.110  /  west-ag .120-.130)

Configure trust — shared root CA

Both clusters share a root CA. Per-cluster intermediates are signed from it. The trust domain is locked to cluster.local on both clusters — the enterprise-agentgateway-waypoint binary hardcodes this and it must match.

Generate root CA

mkdir -p certs

openssl genrsa -out certs/root-ca.key 4096
openssl req -new -x509 -days 3650 \
  -key certs/root-ca.key \
  -subj "/O=Solo Demo/CN=Shared Root CA" \
  -out certs/root-ca.crt

Generate per-cluster intermediates

The SAN spiffe://cluster.local/ns/istio-system/sa/citadel is required — Solo Istio's cross-cluster cert-chain validation checks it.

for CTX in $CLUSTER1 $CLUSTER2; do
  N="${CTX#kind-}"
  openssl genrsa -out certs/${N}-ca.key 4096

  openssl req -new \
    -key certs/${N}-ca.key \
    -subj "/O=Solo Demo/CN=${N} Intermediate CA" \
    -out certs/${N}-ca.csr

  openssl x509 -req -days 3650 \
    -in  certs/${N}-ca.csr \
    -CA  certs/root-ca.crt \
    -CAkey certs/root-ca.key \
    -CAcreateserial \
    -extfile <(printf "subjectAltName=URI:spiffe://cluster.local/ns/istio-system/sa/citadel\nbasicConstraints=CA:TRUE\nkeyUsage=keyCertSign,cRLSign") \
    -out certs/${N}-ca.crt

  # cert-chain = intermediate + root
  cat certs/${N}-ca.crt certs/root-ca.crt \
    > certs/${N}-cert-chain.pem
done

Apply cacerts secrets

for CTX in $CLUSTER1 $CLUSTER2; do
  N="${CTX#kind-}"

  kubectl --context $CTX create namespace istio-system --dry-run=client -o yaml \
    | kubectl --context $CTX apply -f -

  kubectl --context $CTX -n istio-system create secret generic cacerts \
    --from-file=ca-cert.pem=certs/${N}-ca.crt \
    --from-file=ca-key.pem=certs/${N}-ca.key \
    --from-file=root-cert.pem=certs/root-ca.crt \
    --from-file=cert-chain.pem=certs/${N}-cert-chain.pem \
    --dry-run=client -o yaml | kubectl --context $CTX apply -f -
done

Install Gateway API CRDs

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX apply -f \
    https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml
done

Install Solo Istio Ambient — Gloo Operator + ServiceMeshController

The Gloo Operator manages istiod + ztunnel + CNI as a single lifecycle. You declare what you want via a ServiceMeshController CR; the operator reconciles it.

Install Gloo Operator on both clusters

for CTX in $CLUSTER1 $CLUSTER2; do
  helm upgrade --install gloo-operator \
    oci://us-docker.pkg.dev/solo-public/gloo-operator-helm/gloo-operator \
    --kube-context $CTX \
    --namespace gloo-system --create-namespace \
    --version $GLOO_OPERATOR_VERSION \
    --wait
done

Create the Solo Istio licence secret

Must live in istio-system — istiod-gloo reads it via secretKeyRef, which only resolves from the pod's own namespace. The licence JWT itself must have "lt": "ent" for MultiCluster to unlock; trial JWTs (lt: trial, product: gloo-trial) won't satisfy the feature gate even when the secret is wired correctly.

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX -n istio-system create secret generic solo-istio-license \
    --from-literal=license="${SOLO_ISTIO_LICENSE_KEY}" \
    --dry-run=client -o yaml | kubectl --context $CTX apply -f -
done

Apply ServiceMeshController — east

kubectl --context $CLUSTER1 apply -f - <<EOF
apiVersion: operator.gloo.solo.io/v1
kind: ServiceMeshController
metadata:
  name: managed-istio
  namespace: gloo-system
spec:
  cluster: ${CLUSTER1#kind-}
  network: ${CLUSTER1#kind-}
  trustDomain: cluster.local
  version: "${ISTIO_VERSION_OPERATOR}"
  dataplaneMode: Ambient
  distribution: Standard
  scalingProfile: Demo
EOF

Apply ServiceMeshController — west

kubectl --context $CLUSTER2 apply -f - <<EOF
apiVersion: operator.gloo.solo.io/v1
kind: ServiceMeshController
metadata:
  name: managed-istio
  namespace: gloo-system
spec:
  cluster: ${CLUSTER2#kind-}
  network: ${CLUSTER2#kind-}
  trustDomain: cluster.local
  version: "${ISTIO_VERSION_OPERATOR}"
  dataplaneMode: Ambient
  distribution: Standard
  scalingProfile: Demo
EOF

Wait for istiod to be ready

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX -n istio-system wait \
    --for=condition=Available deployment/istiod-gloo \
    --timeout=300s
done

Patch mesh components — required env vars for Ambient peering

The Solo Istio Ambient multicluster manual install guide sets these via platforms.peering.enabled: true in the istiod and ztunnel Helm values, which wires them automatically. The Gloo Operator 0.5.2 SMC schema doesn't expose that flag yet, so we apply the three required vars directly:

• PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES=false on istiod — enables Ambient endpoint rewriting (cross-cluster endpoints resolve via east-west gateway, not K8s WorkloadEntries)
• SOLO_LICENSE_KEY on istiod — sourced from the solo-istio-license Secret. pilot-discovery only reads this exact env var (not a mount, not LICENSE_KEY / GLOO_LICENSE_KEY) — without it the licence stays unread and the multicluster feature gate stays closed
• L7_ENABLED=true on ztunnel — enables L7-aware HBONE tunnelling through waypoints

# istiod — disable K8s WorkloadEntry selection AND wire the SOLO_LICENSE_KEY env
for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX -n istio-system patch deployment istiod-gloo \
    --type=json -p='[
      {"op":"add","path":"/spec/template/spec/containers/0/env/-",
       "value":{"name":"PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES","value":"false"}},
      {"op":"add","path":"/spec/template/spec/containers/0/env/-",
       "value":{"name":"SOLO_LICENSE_KEY",
                "valueFrom":{"secretKeyRef":{"name":"solo-istio-license","key":"license"}}}}
    ]'
done

# ztunnel — enable L7-aware HBONE (required for waypoint traffic across clusters)
for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX -n istio-system patch daemonset ztunnel \
    --type=json -p='[
      {"op":"add","path":"/spec/template/spec/containers/0/env/-",
       "value":{"name":"L7_ENABLED","value":"true"}}
    ]'
done

# Wait for rollouts
for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX -n istio-system rollout status deployment/istiod-gloo --timeout=120s
  kubectl --context $CTX -n istio-system rollout status daemonset/ztunnel --timeout=120s
done

Verify the licence has unlocked multicluster — istiod should now register the global hostname for any Service labelled solo.io/service-scope=global:

kubectl --context $CLUSTER1 -n istio-system logs deploy/istiod-gloo --tail=200 \
  | grep -iE "licens|enterpris|multi.?cluster"
# expect: no "invalid license" / "feature locked" errors

# Once a global Service exists (productpage in the cloud-connectivity lab):
istioctl --context $CLUSTER1 multicluster check | grep "Shared Services"
# expect: ✅ Shared Services Check: ... globally shared services found

Create istiod alias Service

The enterprise-agentgateway-waypoint binary hardcodes CA_ADDRESS=istiod.istio-system.svc:15012 but Gloo Operator names the deployment istiod-gloo.

for CTX in $CLUSTER1 $CLUSTER2; do
  kubectl --context $CTX apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: istiod
  namespace: istio-system
spec:
  selector:
    app: istiod
  ports:
  - name: grpc-xds
    port: 15010
  - name: https-dns
    port: 15012
  - name: https-webhook
    port: 443
    targetPort: 15017
EOF
done

Peer clusters — east-west HBONE gateways

This installs the Istio east-west gateway on each cluster (the HBONE mesh fabric). The enterprise agentgateway waypoint sits on top of this at L7 — it does not replace it.

Label istio-system with the cluster network

Without this label istiod can't classify remote pod networks and endpoint rewriting never fires.

kubectl --context $CLUSTER1 label ns istio-system topology.istio.io/network=${CLUSTER1#kind-} --overwrite
kubectl --context $CLUSTER2 label ns istio-system topology.istio.io/network=${CLUSTER2#kind-} --overwrite

Install east-west gateway — both clusters

for CTX in $CLUSTER1 $CLUSTER2; do
  N="${CTX#kind-}"
  kubectl --context $CTX create namespace istio-eastwest --dry-run=client -o yaml \
    | kubectl --context $CTX apply -f -

  helm upgrade --install peering-eastwest \
    oci://us-docker.pkg.dev/soloio-img/istio-helm/peering \
    --kube-context $CTX \
    --namespace istio-eastwest \
    --version $ISTIO_VERSION \
    --set eastwest.create=true \
    --set eastwest.cluster=$N \
    --set eastwest.network=$N \
    --set remote.create=false \
    --wait
done

Wait for MetalLB to assign LB IPs

EAST_EW_IP=$(kubectl --context $CLUSTER1 -n istio-eastwest \
  get svc istio-eastwest -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
WEST_EW_IP=$(kubectl --context $CLUSTER2 -n istio-eastwest \
  get svc istio-eastwest -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "east-ag east-west GW: $EAST_EW_IP    west-ag east-west GW: $WEST_EW_IP"
# → east-ag east-west GW: 172.22.x.x   west-ag east-west GW: 172.22.x.x

Add remote peer reference — east knows about west

helm upgrade --install remote-peers \
  oci://us-docker.pkg.dev/soloio-img/istio-helm/peering \
  --kube-context $CLUSTER1 \
  --namespace istio-eastwest \
  --version $ISTIO_VERSION \
  --set eastwest.create=false \
  --set remote.create=true \
  --set "remote.items[0].cluster=${CLUSTER2#kind-}" \
  --set "remote.items[0].network=${CLUSTER2#kind-}" \
  --set 'remote.items[0].trustDomain=cluster.local' \
  --set "remote.items[0].address=${WEST_EW_IP}" \
  --set 'remote.items[0].hbonePort=15008' \
  --set 'remote.items[0].xdsPort=15012'

Add remote peer reference — west knows about east

helm upgrade --install remote-peers \
  oci://us-docker.pkg.dev/soloio-img/istio-helm/peering \
  --kube-context $CLUSTER2 \
  --namespace istio-eastwest \
  --version $ISTIO_VERSION \
  --set eastwest.create=false \
  --set remote.create=true \
  --set "remote.items[0].cluster=${CLUSTER1#kind-}" \
  --set "remote.items[0].network=${CLUSTER1#kind-}" \
  --set 'remote.items[0].trustDomain=cluster.local' \
  --set "remote.items[0].address=${EAST_EW_IP}" \
  --set 'remote.items[0].hbonePort=15008' \
  --set 'remote.items[0].xdsPort=15012'

Cross-apply remote secrets (control-plane discovery)

Without these, istiod's "Number of remote clusters" stays at 0 and cross-cluster services never get endpoints.

# East's credentials onto west
istioctl create-remote-secret \
  --context $CLUSTER1 \
  --name ${CLUSTER1#kind-} | \
  kubectl --context $CLUSTER2 apply -f -

# West's credentials onto east
istioctl create-remote-secret \
  --context $CLUSTER2 \
  --name ${CLUSTER2#kind-} | \
  kubectl --context $CLUSTER1 apply -f -

Verify peering

istioctl --context $CLUSTER1 multicluster check

Expected output:

✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
     ✅ istio-eastwest/istio-eastwest available at <MetalLB IP>
✅ Peers Check: all clusters connected
     ✅ Connected to <peer cluster> via <peer LB IP>
ℹ️  Shared Services Check: no globally shared services found      (1 after Step 09)
ℹ️  Intermediate Certs Compatibility Check: root certificate SHA256 sum: <hash>
✅ Network Configuration Check: all network configurations are valid

Install Solo Enterprise agentgateway

Two helm installs per cluster: CRDs first, then the agentgateway (controller + Rust agentgateway proxy). The chart registers both enterprise-agentgateway (ingress) and enterprise-agentgateway-waypoint (L7 / egress) GatewayClasses.

Install CRDs

for CTX in $CLUSTER1 $CLUSTER2; do
  helm upgrade --install agentgateway-crds \
    oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
    --kube-context $CTX \
    --namespace agentgateway-system --create-namespace \
    --version $AGW_VERSION \
    --wait
done

Install Enterprise agentgateway

helm upgrade --install enterprise-agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --kube-context $CLUSTER1 \
  --namespace agentgateway-system \
  --version $AGW_VERSION \
  --set licensing.licenseKey="${AGENTGATEWAY_LICENSE_KEY}" \
  --wait

helm upgrade --install enterprise-agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --kube-context $CLUSTER2 \
  --namespace agentgateway-system \
  --version $AGW_VERSION \
  --set licensing.licenseKey="${AGENTGATEWAY_LICENSE_KEY}" \
  --wait

Label agentgateway-system for Ambient + network topology

Every workload namespace — including agentgateway-system — needs these two labels:

• istio.io/dataplane-mode=ambient — enrolls all pods in this namespace into the Ambient mesh so ztunnel intercepts their traffic. Without it, agentgateway pods sit outside the mesh and can't use HBONE to reach pods in the other cluster.
• topology.istio.io/network=<name> — tells istiod which network these pods belong to. istiod uses this to decide routing: same network → go direct; different network → route via the east-west gateway. Without it, cross-cluster endpoint rewriting never fires.

kubectl --context $CLUSTER1 label namespace agentgateway-system \
  istio.io/dataplane-mode=ambient \
  topology.istio.io/network=${CLUSTER1#kind-} \
  --overwrite

kubectl --context $CLUSTER2 label namespace agentgateway-system \
  istio.io/dataplane-mode=ambient \
  topology.istio.io/network=${CLUSTER2#kind-} \
  --overwrite

Verify GatewayClasses are registered

kubectl --context $CLUSTER1 get gatewayclasses
# Should list enterprise-agentgateway and enterprise-agentgateway-waypoint

Solo Istio Management Plane — Gloo UI (optional)

Deploys the Gloo Platform management plane on $CLUSTER1 with $CLUSTER2 registered as a workload cluster. Gives you a UI with a service graph that visualises the multicluster mesh, plus centralised policy/insights. Mirrors the upstream Gloo Management Plane section, adapted for our east-ag/west-ag cluster naming.

Install meshctl

curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v2.12.0 sh -
export PATH=$HOME/.gloo-mesh/bin:$PATH
meshctl version

Write the management-plane values file

Same shape as the upstream mgmt-values.yaml — just retargeted at east-ag instead of cluster1.

cat > /tmp/mgmt-values.yaml <<'EOF'
common:
  cluster: east-ag
glooAgent:
  enabled: true
  runAsSidecar: true
  relay:
    serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
glooAnalyzer:
  enabled: true
glooMgmtServer:
  enabled: true
  registerCluster: true
  policyApis:
    enabled: true
glooInsightsEngine:
  enabled: true
glooUi:
  enabled: true
prometheus:
  enabled: true
redis:
  deployment:
    enabled: true
telemetryCollector:
  enabled: true
telemetryGateway:
  enabled: true
telemetryGatewayCustomization:
  pipelines:
    traces/jaeger:
      enabled: true
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true
installEnterpriseCrds: false
featureGates:
  ConfigDistribution: false
EOF

Install Gloo Platform on east-ag (management + workload)

helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
  -n gloo-mesh --create-namespace \
  --version=2.12.0 \
  --set installEnterpriseCrds=false \
  --kube-context=$CLUSTER1

helm upgrade -i gloo-platform gloo-platform/gloo-platform \
  -n gloo-mesh \
  --version=2.12.0 \
  --kube-context=$CLUSTER1 \
  --values /tmp/mgmt-values.yaml \
  --set licensing.glooMeshLicenseKey=$GLOO_MESH_LICENSE_KEY

Wait for the telemetry gateway LoadBalancer IP

kubectl --context $CLUSTER1 -n gloo-mesh wait \
  --for=condition=Available deployment/gloo-telemetry-gateway --timeout=300s

export TELEMETRY_GATEWAY_ADDRESS=$(kubectl --context $CLUSTER1 \
  -n gloo-mesh get svc gloo-telemetry-gateway \
  -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}"):4317

echo "telemetry gateway: $TELEMETRY_GATEWAY_ADDRESS"

Register west-ag as a workload cluster

meshctl cluster register west-ag \
  --kubecontext $CLUSTER1 \
  --profiles gloo-mesh-agent \
  --remote-context $CLUSTER2 \
  --telemetry-server-address $TELEMETRY_GATEWAY_ADDRESS

Launch the UI

meshctl dashboard
# Opens http://localhost:8090 in your browser — Gloo UI with a service graph
# of both clusters. Workloads light up once you run a lab that deploys traffic.