Istio Ingress Gateway & Ambient Multicluster Mesh hosted on Kind

What: Clones the kind-specific setup repo (this guide's scripts and configs) and Ram Vennam's upstream ambient-multicluster-workshop side by side.

Why: The kind scripts handle laptop-specific concerns (cluster creation, MetalLB pools) that the upstream workshop skips because it assumes a cloud provider. You'll bounce between the two: kind scripts for steps 2–3, then the upstream README from step 4 onward with the fixes below.

What: Exports two shell variables that name the kind kubeconfig contexts (kind-east and kind-west) for the rest of the lab.

Why: Every kubectl --context=$CLUSTER1 / $CLUSTER2 command in this guide and the upstream workshop reads these. Export them in every new terminal — without them most commands either error out or silently target the wrong cluster.

What: Creates two kind clusters (east and west) using preset configs that lock in non-overlapping pod and service CIDRs (10.10.0.0/16 + 10.96.0.0/16 on east; 10.20.0.0/16 + 10.97.0.0/16 on west).

Why: Ambient multicluster needs each cluster's pod and service CIDRs to be unique so the remote cluster can resolve endpoints without collision. The default kind create cluster would hand both clusters the same CIDRs and break cross-cluster routing.

What: Installs MetalLB v0.14.9 on both clusters and configures each with a distinct IP pool carved out of the kind Docker bridge network (e.g. 172.22.255.200–210 for east, 172.22.255.220–230 for west).

Why: The upstream workshop relies on cloud LoadBalancers to give the Istio east-west gateway a routable IP. On kind there's no cloud LB, so MetalLB plays that role by handing out IPs from the shared Docker bridge — both clusters can then reach each other's east-west gateway directly.

What: Changes into the cloned upstream workshop directory so its relative paths (gloo-extensions.yaml, sample manifests, etc.) resolve correctly.

Why: From here on you're driving the upstream workshop, not this guide's scripts. The "Workshop fixes for kind" section below patches a handful of commands; everything else in Ram's README is run as-is from this directory.

What: Uses AppleScript to open two new Terminal tabs and launches k9s --context=kind-east in one and k9s --context=kind-west in the other.

Why: The whole point of multicluster is watching two clusters react to the same operation. Having both k9s sessions open side by side makes it obvious when, for example, ztunnel on east logs an HBONE connection from a pod on west.

What: Loops over both contexts and applies the gloo-extensions ConfigMap, which sets REQUIRE_3P_TOKEN: false on the Gloo Operator's istiod install.

Why: kind's kubelet only issues first-party service-account tokens. By default istiod's VM ztunnel onboarding flow demands a third-party (audience-scoped) token and rejects the VM. Cloud-managed kubelets (GKE, EKS) issue these natively so the upstream workshop never hits this; on kind you have to relax the requirement explicitly, and it must be in place before istiod first starts.

What: Waits for the istiod pod in each cluster to report condition=ready, then runs istioctl multicluster expose on each context to create the east-west Gateway in istio-gateways.

Why: istioctl multicluster expose queries istiod over XDS to learn the cluster's network name (the value of the topology.istio.io/network label on istio-system). If istiod isn't fully reconciled yet, it returns no network and the command aborts. Waiting on the ready condition first removes that race.

What: The upstream test command — read the LoadBalancer IP / hostname from the bookinfo-gateway-istio Service's status, then curl productpage on that address.

Why: Shown here as context for what fails on kind, not for you to run. The MetalLB IP this resolves to (e.g. 172.22.x.x) lives on the Docker bridge and is unreachable from the macOS host without extra routing. The replacement command below uses kubectl port-forward instead.

What: Port-forwards the bookinfo-gateway-istio Service on east to localhost:8080, then curls /productpage through it.

Why: kubectl port-forward tunnels through the Kubernetes API server, which is published on the host loopback by kind — so it works on macOS regardless of how Docker bridges are routed. This proves the ingress + bookinfo cross-cluster path end to end without depending on bridge-network reachability.

What: Creates an istio-egress namespace, deploys a Gateway with gatewayClassName: istio-waypoint listening on HBONE port 15008, and labels the namespace istio.io/use-waypoint=egress-gateway so traffic from it gets steered through the waypoint.

Why: A waypoint Gateway in egress mode acts as an L7 chokepoint for outbound traffic. It's where you attach AuthorizationPolicy in the next step — only specific identities (e.g. ratings) get allowed out to specific external hosts. Without it, ambient pods can call any external address with no policy enforcement.

What: Registers httpbin.org as a ServiceEntry labelled for the egress waypoint, adds a DestinationRule to upgrade the outbound connection to TLS, and applies an AuthorizationPolicy that allows only the bookinfo-ratings service account to reach it.

Why: The ServiceEntry makes httpbin.org a first-class mesh destination so the egress waypoint can apply policy to it. The DestinationRule originates TLS at the waypoint (the workload talks plain HTTP to :80, the waypoint negotiates HTTPS to :443). The AuthorizationPolicy is the actual control: identity-based allow-listing of who in the mesh is permitted to call this external host.

What: Execs into a ratings pod and curls httpbin.org/get (expected to succeed), then execs into a reviews pod and runs the same curl (expected to be blocked).

Why: Demonstrates that the egress AuthorizationPolicy is enforced at the waypoint based on the caller's SPIFFE identity, not its source IP. Both pods sit in the same namespace on the same network — only the service-account principal differs, and only the allow-listed one gets through.

What: Creates a vm1 namespace and a vm1 ServiceAccount inside it on cluster $CLUSTER1.

Why: The VM's mesh identity comes from a service-account token, not its IP or hostname. Once the VM is enrolled, istiod issues it a SPIFFE identity of the form cluster.local/ns/vm1/sa/vm1, which any AuthorizationPolicy can match. This is what makes a non-Kubernetes workload a first-class citizen of the mesh.

What: An Ubuntu 22.04 image with curl, ca-certificates, iproute2, ping, dig, and the Docker CE engine baked in.

Why: The "VM" needs Docker because ztunnel is shipped as a Docker image, and on a real VM you'd run docker run to start it. Recreating that environment in a container (Docker-in-Docker) keeps the workshop steps identical to the real-VM flow without provisioning actual hardware.

What: Builds the Dockerfile.vm from the previous step and tags the resulting image vm-with-docker.

Why: Local-only image, so no push to a registry needed. The next step references this tag when starting the mock VM container.

What: Removes any previous vm1 container, then starts a fresh one attached to the kind Docker network in --privileged mode. Inside, it boots dockerd with the vfs storage driver and parks the container alive with sleep infinity.

Why: Attaching to the kind network gives the "VM" an IP on the same L2 segment as the cluster nodes and the east-west gateway — same network plane a real on-prem VM would sit on. --privileged is required for Docker-in-Docker; vfs avoids the nested overlay2 failure that bites on most stock setups.

What: Asks Docker for the IP address the vm1 container was given on its attached networks (just one — the kind bridge).

Why: You'll need this IP when you create the WorkloadEntry in VM STEP 7 — it's the address field that tells the mesh where the VM lives. Also a quick sanity check that the container joined the kind network rather than Docker's default bridge.

What: Polls docker info inside the vm1 container every two seconds until it returns successfully.

Why: Nested dockerd takes several seconds to initialise its vfs graph driver on first start. Running the ztunnel docker run before dockerd is ready fails with a confusing "Cannot connect to the Docker daemon" error. This poll blocks until the inner daemon is up.

What: Prints the last 20 lines of the nested dockerd's log file (which the docker run redirected to /var/log/dockerd.log inside the container).

Why: First place to look when the poll above hangs. The most common failure is the overlay2 graphdriver error (fixed by ensuring --storage-driver=vfs made it into the docker run command).

What: Points PATH at the Solo istioctl binary, then runs istioctl bootstrap for the vm1 service account and captures the resulting JWT into $BOOTSTRAP_TOKEN.

Why: istioctl bootstrap is the Solo-extension command that mints a long-lived service-account token plus the istiod address, CA address, trust domain, and east-west GW IP — everything ztunnel needs to join the mesh. Only Solo's istioctl ships this subcommand; the upstream istioctl does not.

What: Opens an interactive bash session inside the vm1 container.

Why: Everything from here until VM STEP 6's "exit" is run inside the mock VM — the ztunnel docker run, the SOCKS5 curl tests, all of it. From the mesh's perspective these commands are being executed on a VM that's joined the mesh, not on the host.

What: Sets the BOOTSTRAP_TOKEN environment variable inside the VM to the value generated on the host in VM STEP 4.

Why: BOOTSTRAP_TOKEN is a separate shell from the host one, so the variable does not propagate automatically. ztunnel reads this env var on startup to authenticate to istiod and pull its initial config.

What: Starts the Solo ztunnel container inside the VM with --network host, --privileged, and the two env vars BOOTSTRAP_TOKEN and ALWAYS_TRAVERSE_NETWORK_GATEWAY=true.

Why: --network host lets ztunnel intercept the VM's actual traffic (rather than only its container namespace's), --privileged is required for the iptables / eBPF redirect rules ztunnel installs, and ALWAYS_TRAVERSE_NETWORK_GATEWAY=true forces outbound mesh traffic through the east-west GW HBONE port instead of attempting a direct pod-to-pod path that wouldn't work from outside the cluster.

What: Greps ztunnel's stdout for the lines that prove it joined the mesh: XDS stream up, license accepted, readiness reached.

Why: If Stream established never appears, the bootstrap token is wrong, expired, or the east-west GW is unreachable. If license shows anything other than OK, the Solo licence on istiod isn't entitled for VM workloads. Surfaces the failure mode quickly without trawling the full log.

What: Exports ALL_PROXY=socks5h://127.0.0.1:15080 so curl sends its requests via ztunnel's local SOCKS5 listener, then calls productpage by Kubernetes short name and by the mesh-internal global hostname.

Why: ztunnel exposes its SOCKS5 listener as the simplest way to drive mesh traffic from a VM — anything routed through it is intercepted, mTLS-wrapped, and tunnelled to the east-west GW. The two hostnames show both local-cluster resolution and global (cross-cluster) failover from the VM's perspective.

What: Labels the vm1 namespace ambient and on network=cluster1, captures the VM's Docker IP, then applies a WorkloadGroup (the VM class), a WorkloadEntry (this specific VM at $VM_IP), and a ServiceEntry that makes vm1.vm1.svc.cluster.local resolve to it.

Why: ztunnel running in the VM gets the mesh's view into the VM; this block does the inverse — registers the VM into the mesh so other workloads can address it as a named service. The topology.istio.io/network label is what tells istiod how to route to it (via east-west GW vs direct), and the WorkloadEntry serviceAccount field anchors the VM's SPIFFE identity for policy.

What: Tails the cluster ztunnel logs from the last 30 seconds and filters for XDS "received response" lines.

Why: Confirms that istiod has pushed the new WorkloadEntry (as an istio.workload.Address) down to every ztunnel pod. If the count never increments, either the WorkloadEntry wasn't admitted (check the namespace's ambient label) or the in-cluster ztunnel isn't subscribing — and cross-mesh traffic to vm1 will be a black hole.

What: Applies an AuthorizationPolicy in the bookinfo namespace with action: DENY, matching the principal cluster.local/ns/vm1/sa/vm1.

Why: Targets only the VM's SPIFFE identity — every other caller (the in-cluster bookinfo services) is unaffected. DENY takes precedence over any ALLOW, so this is the right shape for "this workload must never reach these services" regardless of what else is in the policy graph. No firewall rules, no app changes — identity-based segmentation.

What: From inside the VM, curls productpage through ztunnel's SOCKS5 proxy — the same call that worked in VM STEP 6.

Why: Proves the DENY took effect at the dataplane: ztunnel on the destination side rejects the inbound HBONE with RBAC: access denied because the source identity matches the deny rule. The traffic never reaches the productpage pod — enforcement happens at the mesh edge.

What: Deletes the deny-vm1 AuthorizationPolicy from the bookinfo namespace.

Why: Restores the cluster to its known-good baseline before the traffic-simulation section runs. The next steps assume vm1 can reach productpage again.

What: Launches a long-running curl pod (vm1-load) in the bookinfo namespace that calls vm1.vm1.svc.cluster.local:9080 once a second.

Why: Generates a steady cross-workload edge (pod → VM via the ServiceEntry) so the Gloo Platform graph has live metrics to render. Without ongoing traffic the dashboard edges go grey within a few seconds.

What: Deletes the vm1-load pod that was producing the steady curl traffic.

Why: Stops the synthetic load so the cluster goes back to idle. The pod was started with kubectl run (not a Deployment) so a single delete tears it down for good.

What: Resolves the current ratings pod name, then execs into it and curls httpbin.org/get through the egress waypoint.

Why: Drives a live edge on the Gloo graph from ratings through egress-gateway out to httpbin.org — the workload identity matches the allow rule from earlier, so it succeeds and shows up as a green edge.

What: Same shape as the previous command, but execs into a reviews pod instead of ratings and curls the same external host.

Why: The AuthorizationPolicy only names bookinfo-ratings; reviews is implicitly denied. The call gets blocked at the egress waypoint, which surfaces as a red/denied edge on the Gloo graph — visual proof that egress policy is identity-aware, not network-aware.