Air-gapped image management with containerd registry mirrors

Why not just override the Helm values?

Both approaches end up pulling from registry.internal. The difference is how many surfaces you have to touch and how many places a miss can hide.

Mechanics either way: skopeo copy --all docker://upstream/path:tag docker://registry.internal/path:tag preserves digest, multi-arch manifest list and tag — only the hostname changes. From there:

What this pattern actually does

The cluster is already in a tightly controlled air-gapped environment — no public-internet path, vetted ingress, audited change control. The mirror pattern doesn't replace any of that; it's a routing control that makes the rest of the install behave correctly inside that environment. Be explicit about what the mirror gives you, and which of the deeper sections cover everything else.

• What the mirror does: redirects every image pull from us-docker.pkg.dev / gcr.io / docker.io to registry.internal, transparently. Charts and CRs install unmodified; references stay canonical so SBOMs, Cosign signatures and audit logs all line up with what Solo published.
• What the rest of the article adds on top:
  • Provenance — Cosign verify on the pull path (Step 2), platform-signed transfer manifest (Step 2), admission-time verify (Step 6).
  • Air-gap enforcement — egress firewall + bypass probe so public registries are demonstrably unreachable, not just assumed unreachable (Step 7).
  • Credential handling — TLS / mTLS / private CA, file permissions, secret-injection patterns; the goal is no long-lived plaintext token on any node (Step 4).
  • Bootstrap hygiene — Job-per-node with TTL instead of a perpetual privileged DaemonSet, PSA + NetworkPolicy on the bootstrap namespace (Step 5).
  • Availability — mirror HA, node-local pre-pull so a brief mirror outage during a rollout doesn't break running pods (HA & DR).

Where the registry lives — and the catch-22

The mirror registry must be reachable before the cluster needs to pull its first image. Two patterns — pick the first one unless an explicit constraint forces otherwise.

What lives where

Two clean tiers: a private registry outside the cluster, and a small set of files on every cluster node that point containerd at it. Nothing else changes.

Pick your delivery path

Three mechanisms put the same mirror configuration onto every node. Option 1 is the underlying containerd config the whole article documents; Options 2 and 3 are convenience layers that generate it automatically for CAPI and OpenShift clusters respectively. The image-set and registry mechanics in Steps 1 and 2 below are identical in every path — only the delivery layer changes.

All three options ultimately produce the same on-node containerd config. Option 1 is what this article focuses on; Options 2 and 3 are convenience layers that generate the same config automatically for CAPI and OpenShift clusters.

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata: { name: solo-airgap-workers, namespace: default }
spec:
  template:
    spec:
      files:
      - path: /etc/containerd/certs.d/us-docker.pkg.dev/hosts.toml
        owner: root:root
        permissions: "0644"
        content: |
          server = "https://us-docker.pkg.dev"
          [host."https://registry.internal"]
            capabilities = ["pull", "resolve"]
            ca = "/etc/containerd/certs.d/mirror-ca.crt"
      - path: /etc/containerd/certs.d/mirror-ca.crt
        owner: root:root
        permissions: "0644"
        contentFrom:
          secret: { name: mirror-ca-bundle, key: ca.crt }
      preKubeadmCommands:
      - "systemctl restart containerd"

# Digest pulls — safe by default, immutable identity
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata: { name: solo-airgap-digest }
spec:
  imageDigestMirrors:
  - source:  us-docker.pkg.dev/solo-public/enterprise-agentgateway
    mirrors: [registry.internal/solo-public/enterprise-agentgateway]
    mirrorSourcePolicy: NeverContactSource
  - source:  us-docker.pkg.dev/solo-public/istio
    mirrors: [registry.internal/solo-public/istio]
    mirrorSourcePolicy: NeverContactSource
  - source:  docker.io/library
    mirrors: [registry.internal/dockerhub-library]
    mirrorSourcePolicy: NeverContactSource
---
# Tag pulls — needed for Solo Helm charts
apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata: { name: solo-airgap-tag }
spec:
  imageTagMirrors:
  - source:  us-docker.pkg.dev/solo-public/enterprise-agentgateway
    mirrors: [registry.internal/solo-public/enterprise-agentgateway]
    mirrorSourcePolicy: NeverContactSource

export VER=2026.5.0

# Pull every chart you intend to install
helm pull oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --version $VER --untar
helm pull oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
  --version $VER --untar
# Repeat for kgateway, istio, kagent, gateway-api CRDs, etc.

# Render with realistic values and extract every image reference
helm template enterprise-agentgateway ./enterprise-agentgateway \
  -f values.yaml \
  | yq -r '.. | .image? // empty' \
  | grep -v '^$' | sort -u > images.txt

kubectl get configmap -n agentgateway-system -o yaml \
  | grep -E 'image:|repository:' | sort -u >> images.txt

# Direct copy if the connected host can reach both upstream and the air-gap registry
while read img; do
  src="docker://$img"
  dst="docker://registry.internal/${img}"   # keep the upstream path
  skopeo copy --all "$src" "$dst"
done < images.txt

# Two-step copy if the connected host has no path to the air-gap network
# 1. On the connected side: copy to a directory
skopeo copy --all docker://us-docker.pkg.dev/solo-public/.../agentgateway-enterprise:$VER \
  dir:/transfer/agentgateway-enterprise-$VER
# 2. Sneakernet the directory across the gap
# 3. On the air-gap side: push from the directory to the registry
skopeo copy --all dir:/transfer/agentgateway-enterprise-$VER \
  docker://registry.internal/solo-public/.../agentgateway-enterprise:$VER

# Verify, then copy. Fail closed.
while read img; do
  cosign verify --key https://solo.io/cosign.pub "$img" > /dev/null \
    || { echo "REJECTED: $img"; exit 1; }
  skopeo copy --all "docker://$img" "docker://registry.internal/$img"
  cosign copy   "$img" "registry.internal/$img"        # carry the signature
done < images.txt

# Build the digest inventory, sign it
while read img; do
  digest=$(skopeo inspect --no-tags --format '{{.Digest}}' "docker://registry.internal/$img")
  printf '%s@%s\n' "${img%:*}" "$digest"
done < images.txt | sort -u > transfer-manifest-$VER.txt

cosign sign-blob --key platform-team.key transfer-manifest-$VER.txt > transfer-manifest-$VER.sig

# On the air-gap side, before any helm install:
cosign verify-blob --key platform-team.pub \
  --signature transfer-manifest-$VER.sig transfer-manifest-$VER.txt

version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

sudo systemctl restart containerd

/etc/containerd/certs.d/
├── us-docker.pkg.dev/hosts.toml
├── gcr.io/hosts.toml
└── docker.io/hosts.toml

server = "https://us-docker.pkg.dev"   # upstream fallback; unreachable in air-gap, kept for clarity

[host."https://registry.internal"]
  capabilities = ["pull", "resolve"]
  # If the mirror uses a private CA:
  ca = "/etc/containerd/certs.d/us-docker.pkg.dev/mirror-ca.crt"
  # Keep override_path off so containerd preserves the upstream path on the mirror
  # (set to true only if your mirror flattens everything under one project)
  # override_path = true

server = "https://registry-1.docker.io"

[host."https://registry.internal"]
  capabilities = ["pull", "resolve"]

server = "https://gcr.io"

[host."https://registry.internal"]
  capabilities = ["pull", "resolve"]

# mTLS example
[host."https://registry.internal"]
  capabilities = ["pull", "resolve"]
  ca     = "/etc/containerd/certs.d/us-docker.pkg.dev/mirror-ca.crt"
  client = [["/etc/containerd/certs.d/us-docker.pkg.dev/client.crt",
             "/etc/containerd/certs.d/us-docker.pkg.dev/client.key"]]

# /etc/containerd/config.toml — MUST be 0600 root:root
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.internal".auth]
  username = "robot$airgap-puller"
  password = "<token>"

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-solo-images
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: verify-solo-cosign
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaces:
          - agentgateway-system
          - kgateway-system
          - istio-system
          - kagent-system
    verifyImages:
    - imageReferences:
      - "registry.internal/solo-public/*"
      attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              <Solo's cosign.pub here>
              -----END PUBLIC KEY-----
      mutateDigest: true   # rewrite tag→digest so pods can't drift
      verifyDigest: true
      required: true

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: verify-solo-images
spec:
  images:
  - glob: "registry.internal/solo-public/**"
  authorities:
  - key:
      data: |-
        -----BEGIN PUBLIC KEY-----
        <Solo's cosign.pub here>
        -----END PUBLIC KEY-----

# Default-deny egress for the namespaces that host bootstrap / pre-pull pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
  namespace: airgap-bootstrap
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress: []

---
# Then explicitly allow the mirror + DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-mirror-egress
  namespace: airgap-bootstrap
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - ipBlock:
        cidr: 10.50.0.0/24   # registry.internal subnet
    ports:
    - protocol: TCP
      port: 443
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

# Schedule as a CronJob — alerts if the probe ever succeeds
apiVersion: batch/v1
kind: CronJob
metadata:
  name: airgap-bypass-probe
  namespace: airgap-bootstrap
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: probe
            image: registry.internal/utils/curl:8.5
            command: ["/bin/sh","-c"]
            args:
            - |
              # If ANY of these succeed, the air-gap is leaking
              for h in us-docker.pkg.dev gcr.io registry-1.docker.io quay.io; do
                if curl --max-time 5 -sSf "https://$h/v2/" > /dev/null 2>&1; then
                  echo "LEAK: $h reachable from cluster"
                  exit 2
                fi
              done
              echo "ok — no public registries reachable"

Upgrades and image lifecycle

The hard case is the rolling upgrade — both v1 and v2 images get pulled from the mirror concurrently for hours to days. Three rules:

1. Serve both N and N-1 for the full rollout. Push v2, immediately GC v1, and any restart / reschedule / rollback on a not-yet-drained node ImagePullBackOffs. The most common air-gap upgrade failure.
2. Surge upgrades amplify the overlap window (maxSurge > 0 or rolling-replacement node pools — new node up before old node drains, both versions pulling at once).
3. Istio sidecar mode needs its own retention rule — see below.

Istio sidecar retention

Sidecar-mode Istio: existing app pods keep running the v1 sidecar until the application pod is restarted. Any v1-sidecar pod that gets rescheduled, OOM-killed, evicted or drained pulls v1 from the mirror — if you've already GC'd v1, it ImagePullBackOffs. Mirror v1 until every workload has restarted on v2. Probe:

# Pods still running v1 — must return empty before GC'ing v1
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{range .spec.containers[?(@.name=="istio-proxy")]}{.image}{"\n"}{end}{end}' \
  | grep ':1.25.0$'

Ambient (ztunnel) avoids this — ztunnel is per-node and rolls with the node, so the retention window matches the node rollout instead of the application-restart cadence.

Retention policy

• N (current): always.
• N-1: always during rollout; never delete during an active rollout (rollback depends on it).
• N-2: through the soak period (typically 2–4 weeks after N is fully rolled).
• Older: archive to cold storage rather than delete — storage is cheap, audit trail + CVE forensics are valuable.

Harbor and Quay both enforce retention by tag pattern + count + age. Configure once and stop hand-managing.

Pre-flight: enumerate the new version

helm pull oci://us-docker.pkg.dev/solo-public/.../enterprise-agentgateway \
  --version $NEW_VER --untar -d ./new
helm template ./new/enterprise-agentgateway -f values.yaml \
  | yq -r '.. | .image? // empty' | sort -u > images-new.txt

# Diff by digest, not by repo:tag — a bumped tag on the same repo
# is a different artefact and a repo:tag diff would miss it.
resolve_digests () {
  while read ref; do
    digest=$(skopeo inspect --no-tags --format '{{.Digest}}' "docker://$ref")
    printf '%s@%s\n' "${ref%:*}" "$digest"
  done
}
resolve_digests < images-new.txt     | sort -u > images-new-digests.txt
resolve_digests < images-current.txt | sort -u > images-current-digests.txt
comm -23 images-new-digests.txt images-current-digests.txt > images-to-mirror.txt

Mirror images-to-mirror.txt before the Helm release. The Helm upgrade is then a no-network operation.

Pre-pull onto nodes (optional but worth it)

For large clusters or fragile mirror links: warm every node's local content store before rolling, so the rollout itself doesn't depend on registry availability. A privileged Job-per-node with ttlSecondsAfterFinished is cleaner than a perpetual DaemonSet — same effect, no lingering primitive (see Step 5 hardening).

# Per-node Job — runs once, deletes itself
apiVersion: batch/v1
kind: Job
metadata: { name: prepull-$NODE, namespace: airgap-bootstrap }
spec:
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: Never
      nodeName: $NODE
      containers:
      - name: prepull
        image: registry.internal/utils/crictl:1.30   # signed, pinned by digest
        command: ["/bin/sh","-c","for i in $IMAGES; do crictl pull $i; done"]
        env: [{ name: IMAGES, value: "registry.internal/...:$NEW_VER ..." }]
        securityContext: { privileged: true }
        volumeMounts: [{ name: crisock, mountPath: /run/containerd/containerd.sock }]
      volumes: [{ name: crisock, hostPath: { path: /run/containerd/containerd.sock } }]

What does NOT change at upgrade time

The mirror config on each node (hosts.toml / IDMS / registries.yaml) maps upstream→mirror, not image→mirror — only the mirror's contents change at upgrade. The exception is a new upstream registry (e.g. Solo adds a new GAR project) — that is a node config change, push it via the same delivery mechanism you used for the initial setup.

Digest pinning (high-assurance only) & rollback

Tags can be retagged; digests can't. If immutability matters, resolve every tag to @sha256: at mirror time and pin Helm/manifest references by digest — accept noisy upgrade diffs as the cost of supply-chain integrity.

Rollback works iff N-1 is still in the mirror — helm rollback recovers because v1 images remain in registry.internal and in most nodes' local content stores (containerd doesn't GC recently-run images).

HA and DR for the mirror

Every node depends on the mirror for every pull. "One per air-gap network is enough" is right for steady state, wrong for the failure modes that page someone.

• Topology: single instance (lab only) · active-passive with replicated blob store (most production air-gaps) · active-active behind LB on shared object storage (multi-cluster). Layer the multi-mirror trick in hosts.toml on top — list registry-primary.internal and registry-secondary.internal in declared order; containerd tries them in order.
• Backup both stores: blob store (S3-compatible — versioning + lifecycle) and metadata DB (Harbor Postgres / Zot embed / Artifactory). Restore in lockstep.
• RTO/RPO targets in writing: RTO-read typically 5–15 min on hot standby; RTO-write hours-tolerable; RPO zero with sync replication, daily snapshots typical for object-storage layouts.
• Behaviour when unreachable: image already on the node → runs (local cache); new pull → fails fast (the mirror is configured to fail closed in air-gap — no upstream fallback attempt); flaky mirror + imagePullPolicy: Always → tail-latency balloon.
• The actual saving grace: node-local pre-pull (Pattern B / DaemonSet warmer) means the mirror only matters for new images and scale-out. Time the pre-pull before any production rollout.

Observability of the mirror itself

The verification step uses journalctl grep. That's fine on day one for one node. On a 200-node cluster you want this running continuously, with alerts.

Instrument: mirror hit/miss ratio (registry metrics) · pull failure rate by image (registry 4xx/5xx) · pull latency p50/p95/p99 by node (probe + access logs) · containerd attempts not hitting the mirror (journalctl / Falco / eBPF) · mirror disk utilisation · CA expiry probe · admission verification failures (Kyverno / policy-controller).

Minimum alerts (Sev-1 paging unless noted): p99 pull latency > 5 s for 5 min · pull failure rate > 1 % for any image (ticket) · any non-mirror outbound 443 from a cluster node · mirror disk > 80 % · CA expiry < 30 days (ticket) · admission policy rejection of a Solo image.

Verification

A green pull via crictl plus pods Running with their upstream image references is the success signal: the runtime-layer mirror is doing its job and the application layer is unchanged.

Getting onto a node without SSH

Most platform teams don't SSH to nodes any more. Use kubectl debug node/... to drop a privileged pod onto the node and chroot /host into the node filesystem:

# Spawn a debug pod scheduled on a specific node
kubectl debug node/<node-name> -it --image=registry.internal/utils/busybox:1.36 -- chroot /host

# Once inside the node fs, run the steps below — crictl, journalctl,
# /etc/containerd/certs.d are all available as if you SSH'd in.

# Alternative: nsenter into containerd's PID namespace via a privileged pod
kubectl run nsenter --rm -it --restart=Never \
  --image=registry.internal/utils/busybox:1.36 \
  --overrides='{"spec":{"hostPID":true,"containers":[{"name":"x","image":"registry.internal/utils/busybox:1.36","stdin":true,"tty":true,"command":["nsenter","--target","1","--mount","--uts","--ipc","--net","--pid","--","sh"]}]}}'

Scope the debug pod's namespace to one that's covered by a Pod-Security privileged profile — by default, kubectl debug node/... creates the pod in default, which on a hardened cluster will be rejected by admission.

# 1. Confirm hosts.toml is in place on a representative node
ls /etc/containerd/certs.d/
cat /etc/containerd/certs.d/us-docker.pkg.dev/hosts.toml

# 2. Force a pull through containerd (bypasses kubelet caching)
sudo crictl pull us-docker.pkg.dev/solo-public/enterprise-agentgateway/agentgateway-enterprise:2026.5.0

# 3. Confirm it actually hit the mirror, not the upstream
sudo journalctl -u containerd --since "5 minutes ago" \
  | grep -E 'registry.internal|us-docker.pkg.dev'
# Expect lines referencing registry.internal; no outbound 443 attempts to us-docker.pkg.dev

# 4. Install Solo charts with their stock values — no image overrides
helm install enterprise-agentgateway-crds \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds \
  --version $VER -n agentgateway-system --create-namespace

helm install enterprise-agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --version $VER -n agentgateway-system

# NOTE: helm does NOT use containerd's hosts.toml for OCI pulls.
# helm uses oras-go and reads ~/.config/helm/registry/config.json
# + the system trust store. Configure helm registries independently,
# or — simpler — point helm install directly at the mirror:
#
#   helm install enterprise-agentgateway \
#     oci://registry.internal/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
#     --version $VER -n agentgateway-system

# 5. Verify pods come up with their original image references intact —
#    they should, because the mirror is invisible at the manifest layer
kubectl get pods -n agentgateway-system -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{"\n"}{end}{end}'
# Expect images like us-docker.pkg.dev/... — and pods Running, not ImagePullBackOff

Pre-upgrade verification

Before bumping a Helm release, confirm both the current and the new image tags resolve through the mirror — on the same node, in sequence. This catches the most common upgrade failure (new images not mirrored yet) before it surfaces as ImagePullBackOff on a half-drained node.

# Current version still resolves
sudo crictl pull us-docker.pkg.dev/solo-public/.../agentgateway-enterprise:$CURRENT_VER

# New version resolves (proves mirror push succeeded)
sudo crictl pull us-docker.pkg.dev/solo-public/.../agentgateway-enterprise:$NEW_VER

# Optionally: every new image from images-to-mirror.txt
xargs -a images-to-mirror.txt -I{} sudo crictl pull {}

Only proceed with helm upgrade after every line in images-to-mirror.txt pulls clean.

Related

• Upgrade Solo Enterprise for Agentgateway — apply this mirror config first, then upgrade with stock Helm values.
• containerd registry configuration reference — full hosts.toml field reference.
• OpenShift ImageDigestMirrorSet / ImageTagMirrorSet — config.openshift.io/v1 reference. OpenShift only; the Machine Config Operator applies it. No equivalent CRD or controller in upstream Kubernetes.
• skopeo copy — multi-arch and digest-preserving image transfer.

Treat this article as a reference shape — the runtime-layer mirror pattern is sound, and the Solo-specific image enumeration step in Step 1 is the one to validate against your install before promoting to production.