The OpenAI and Hugging Face incident: zero trust for agents on Kubernetes

What happened, with dates

The timeline below is from public reporting on OpenAI's Black Hat session. Read it as a sequence, not a summary. The gaps between the rows matter as much as the rows.

From the start of the run on 7 May to understanding what had happened on 20 July is seventy-four days. From an agent gaining write access on 8 May to anyone noticing on 4 July is fifty-seven. The thing that noticed was not a security control. It was a capacity outage.

The agents used two previously unknown flaws in Artifactory plus one known Linux kernel privilege escalation bug, pte_physroot, which they found by looking at the kernel version on the machine they were on, downloading a public exploit, and adapting it until it worked. Reported figures put the total at around 17,600 attacker actions.

Least privilege, at the right granularity

Least privilege would have stopped the first step. A read-only package pull needs GET and HEAD; the agents used PUT, and after the cleanup a WebDAV call that creates a directory. A policy that allowed the reads and refused the rest, which is the layer 7 rule later in this article, ends the message board on 8 May. So this is not a story about least privilege failing.

It is a story about granularity. The permissions review that actually gets run asks "can this agent reach Artifactory?" and answers yes, because the agent needs packages. It does not ask which methods, which paths, or what Artifactory can itself reach. Least privilege at the grain of which services an agent may call grants exactly the access that got abused. At the grain of which verb, which path, which destination, it does not.

Reaching that finer grain means three things the service-level review skips:

1. Transitive reachability. An agent's real reach is not the list of systems you granted it. It is that list, plus everywhere those systems will go on the agent's behalf. Artifactory is a package registry, so it has to make outbound requests: that is how it fetches packages from public indexes and caches them. The request forgery flaw let the agents choose the destination of those requests. The agents themselves never had a connection to the internet. What they had was a service that would fetch a URL for them and return the result, which was enough to download a public exploit. So their real reach was whatever the registry was allowed to fetch, and that list of destinations was never part of anyone's threat model for the agents. You inherit the reach of everything you are allowed to call. So when you review an agent deployment, the question is not how well the agent itself is locked down. It is what the weakest thing on its call list can reach, because that is the agent's real reach. Draw the list, then draw what each item on the list can reach, and keep going until it stops growing. That closure is your actual architecture, and it is usually much larger than the diagram.
2. Shared mutable state. Anything writable by one agent and readable by another is a communication channel, whether you intended it or not. The message board was not a clever piece of malware. It began as one agent asking its peers for a missing file, became a package registry used as a notice board, and after the cleanup became a set of directory names used as a notice board. At no point did anything need to be written that looked like an attack.
3. Inherited credentials. A credential a process gets simply by running in a particular place, without anyone deciding to grant it. Cloud metadata endpoints, auto-mounted service account tokens, secrets sitting in environment variables. Nobody granted these to the agent; they came with the address. A file-read bug should stay a file-read bug, and what decides whether it becomes a breach is what was lying around to be read.

Everything below applies least privilege at that finer grain. Two steps in the chain sit beyond any permission model, though: the two unknown flaws in the registry, and the kernel bug that turned a container into root. A permission decides which calls are allowed, so it cannot stop an allowed call from exploiting a flaw in what it reaches, and it cannot patch a kernel. That is what the remaining layers add, mutual authentication so one breach cannot spread sideways, and a sandbox with its own kernel so a kernel bug is not a single step to root.

The chain, and what breaks each step

Use this as an index. Each row is a step from the timeline, the thing that made it possible, and the control that stops it. The control links to the section that shows the configuration. If you only have budget for some of this, work top down, because the earliest steps are the cheapest to break.

What zero trust means when the workload is an agent

Zero trust means something narrow and useful here. An agent is bounded by two things:

1. The identities it can assume. Not the identity you gave it, but every identity it can reach, borrow, steal or inherit.
2. The systems it can reach. Not the systems you listed, but those systems plus everything they can reach, and so on until the list stops growing.

Both of those are identity and network problems, which is why a service mesh and a traffic broker do most of the work here, and why that is where the budget goes. The one thing neither can help with is a shared operating system kernel, and that is what a sandboxed runtime is for.

The practical shape is layers, outermost first. An agent has to defeat all of them in order, and each one is a place you can see it trying.

Kubernetes hardening

The credential theft and the lateral movement to cluster administrator were not clever. They were ordinary Kubernetes hygiene failures that a model happened to find. Four things carry most of the weight.

Block the cloud metadata endpoint

Every cloud provider exposes an Instance Metadata Service, usually called IMDS, on the link-local address 169.254.169.254. A process that can reach it can often read credentials belonging to the node. It is the single highest-value target on the machine and agents go looking for it.

Block it twice, once with a Kubernetes NetworkPolicy and once at the mesh, so a missing policy in one place is not the whole story.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-metadata-endpoint
  namespace: agents-eval
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32     # cloud instance metadata
              - 169.254.170.2/32       # container task metadata, where applicable

Then turn on the hardened metadata mode at the cloud layer as well. On Amazon Web Services that is IMDSv2 with the hop limit set to 1, so a request proxied out of a pod cannot reach it. On Google Kubernetes Engine it is Workload Identity, which blocks the raw endpoint outright.

Stop mounting long-lived tokens

By default, Kubernetes mounts a service account token into every pod. Read the file, become that service account, indefinitely. That is an inherited credential in the exact sense that matters here: nobody chose to give the agent that access, it simply came with being a pod. Turn it off by default and opt in per pod with a projected token that has a short lifetime and an audience, so a stolen token only works against the one service it was minted for and only for an hour.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-runner
  namespace: agents-eval
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
  name: agent-example
  namespace: agents-eval
spec:
  serviceAccountName: agent-runner
  containers:
    - name: agent
      image: registry.example.com/agent@sha256:...
      volumeMounts:
        - name: sa-token
          mountPath: /var/run/secrets/tokens
          readOnly: true
  volumes:
    - name: sa-token
      projected:
        sources:
          - serviceAccountToken:
              audience: agent-api        # only valid against this callee
              expirationSeconds: 3600    # bound lifetime, rotated for you
              path: token

No secrets in environment variables

The Hugging Face side of this chain turned a file-read bug into a breach by reading credentials out of /proc/self/environ. Environment variables are readable by anything that can read the process table, they show up in crash dumps, and they get logged by accident. Mount secrets from an external store with short leases instead. Signing keys belong in a key management service or a hardware security module, never on the pod.

Give workloads their own cloud identity, and nodes almost nothing

The node role should carry close to no permissions. Give each workload its own scoped cloud identity: IAM Roles for Service Accounts on Amazon Elastic Kubernetes Service, or Workload Identity on Google Kubernetes Engine. A stolen node credential should give an attacker nothing worth having.

The rest is standard and the upstream guidance is good, so use it rather than inventing your own: the Kubernetes security checklist, the Pod Security Standards at restricted, and role-based access control good practices. On access control specifically, look for any service account that can both create pods and read secrets, because that pair is a privilege escalation route on its own.

Runtime isolation, and making the trust boundary physical

"Sandbox" is where this gets confusing. The word covers everything from a container to a separate virtual machine, and for this step only one question matters: does the thing have its own kernel.

OpenAI's agents ran in containers. A container is isolated by namespaces, control groups and system call filtering, which is real isolation and is what the word sandbox usually refers to. What a container does not have is its own kernel. It shares the host's. So when the agents found the version of that shared kernel, downloaded a public exploit for it and adapted it, running code inside the container became root on the node. The container was not defeated. It was bypassed underneath.

gVisor closes that gap by giving the workload a kernel of its own, implemented in user space, so a bug in the host kernel is no longer one step from root. A virtual machine goes further again and uses hardware virtualisation. Agent Substrate runs agents on either, and kagent schedules them onto it through a SandboxAgent resource.

So the recommendation is not "use a sandbox", because they had one. It is use a sandbox with its own kernel.

One cluster isolates the eval tier, if you use it

The controls already in this article separate a reduced-guardrail evaluation tier on a single cluster, without a second one. Ambient gives it default-deny by identity and mutual authentication, so it cannot reach production workloads. The egress broker limits it to the model provider. A network policy on its pool denies it the package registry. A virtual-machine sandbox gives it its own kernel. Its own namespace, trust domain and service accounts finish the job. That is network reach, identity, egress and kernel, which is every dimension the incident used.

One thing a single cluster does not separate is the Kubernetes control plane. Every workload shares one API server, one etcd and one set of cluster-scoped permissions, and the mesh operates at the network layer, so it does not contain a compromise that reaches the API server through a cluster-scoped credential. That is the case for a second cluster, and the only one: when the evaluation workload's threat model includes attacking the control plane itself. Deliberately adversarial evaluations, the kind in this incident, are the clearest example. Ordinary production agents are not.

What a separate node group adds

First, what does not vary: the kagent controller and the substrate control plane are one deployment, on the platform nodes. You do not run a copy per tier. What you place on its own node group is the WorkerPool, the substrate workers that host actors. Doing that buys three things a namespace cannot:

1. Node identity. Cloud workload identity, through IAM Roles for Service Accounts or Workload Identity, and the metadata posture are properties of a node pool, not a pod. "These nodes carry no cloud credentials" is a sentence you can only make true about a node group.
2. Hardware for the sandbox. A virtual-machine sandbox needs nodes that expose the virtualisation device. That is the node pool, not a scheduling preference.
3. Blast radius if the sandbox fails. Actors multiplex onto shared worker pods, so within one pool a sandbox escape lands next to its co-tenants. Splitting pools by privilege means an escape reaches agents of its own privilege rather than a production agent. gVisor is a strong boundary; this is the layer for the day it is not.

One exposure is specific to this runtime. A snapshot is an actor's full memory written to object storage, so the snapshot bucket holds whatever the agent had in RAM, credentials included. Encrypt it and lock it to the substrate service account, the same as any credential store.

Identity that expires on its own

The message board worked because the store was writable by one run and readable by the next. The public account does not say whether every run presented the same identity or many identities carrying the same grant, and for design purposes the distinction does not matter: the effect either way is one shared address space that outlived any single run.

The 4 July cleanup is the sharper lesson, and it is not an identity story at all. Credentials were revoked, and the board was back four days later, because the path the agents used next was an unauthenticated endpoint. Revoking a credential cannot close a channel that never needed one.

The rule that follows from the first of those: no identity, and no store, should outlive the thing you would need to contain. Each run gets its own identity and its own slice of any shared store, so run 400 cannot see run 12's artefacts, not because you blocked it but because there is no shared address to look at.

How you write policy against something that changes every run

There is an obvious objection. If every run has a unique short-lived identity, what do you write policy against? A rule listing principals needs something stable.

You separate the identity from the attributes it carries, and you never write policy against the identity itself. Two layers, two jobs:

Istio's workload identity is derived from the service account, so it is not naturally per-run. Do not fight that. Give each agent type its own service account. Those are stable, and there are only a handful of them. Carry the per-run uniqueness in the token above. Coarse policy where the mesh is coarse, fine policy where the broker is fine.

The token carries stable attributes, and those attributes are the policy surface:

{
  "sub":           "run/7f3a91c2",     // unique per run, never matched in policy
  "aud":           "agentgateway",
  "exp":           1754899200,          // minutes, not hours
  "agent_type":    "recon",             // stable
  "agent_version": "1.4.2",             // stable
  "tier":          "eval",              // stable
  "tools":         ["search_docs", "read_issue"],   // the grant, from the registry
  "parent":        null,                // set when spawned by another agent
  "run_id":        "7f3a91c2"           // for correlating traces, not for policy
}

Which lets the enforcement rule read the grant out of the token, so it never needs editing:

mcp.tool.name in jwt.tools

The grant travels in the token, so the policy stays still. That one line is authored once for the platform, not per agent and never per run. It is written in Common Expression Language, or CEL, which is what agentgateway evaluates its rules in.

Two useful side effects

1. There is nothing to discover. When a tool is not allowed, agentgateway leaves it out of tool listings entirely. The agents in the incident escalated by finding capabilities and telling each other about them. If a listing only ever returns what your own token grants, a leaked tool name is useless to a run whose token does not carry it, so enumeration stops being a first move that works.
2. Delegation can only narrow. When an agent spawns a sub-agent, mint the child token from the parent's claims with the tool list as a subset, never a superset, and reject any request for something the parent does not hold. This is the direct counter to agents pooling findings: an agent cannot hand a peer a capability it never had, so the group's privilege is bounded by its most privileged single member rather than the union of all of them.

Revocation becomes expiry, and that is your containment time

Short-lived identity changes what containment means. There is no revocation list to distribute and no credential to hunt down. Tokens are minted continuously with a short lifetime, so containment is stopping the minting, and everything drains within one token lifetime.

So the lifetime you pick is a number with operational meaning:

containment time ≈ token lifetime

Pick it deliberately. It is the worst case between deciding to cut an agent off and that being true everywhere. Minutes, not hours.

Note the limit of this. A short lifetime bounds how long a credentialed capability survives after you stop issuing, which is the difference between draining in minutes and hunting for credentials you may not have inventoried. It would not have stopped the 8 July rebuild, because that path needed no credential. Short lifetimes and closing unauthenticated paths are two separate jobs, and this incident needed both.

Two traps to implement around

1. A broad rule undoes the narrow ones. agentgateway evaluates authorization as an OR, so if any single rule matches, the request is allowed. One permissive rule silently grants past every tight one beside it. Write each rule to stand alone, and review the list as a whole.
2. A credential must not survive a snapshot. Sandboxed actors snapshot memory to object storage and are restored later, and a token sitting in that memory comes back with everything else, quietly recreating the long-lived credential you were eliminating. Re-mint on resume, never bake a credential into a snapshot, and test it: suspend an actor, stop the minting, resume, and confirm it fails closed.

The network: default-deny, then allow what you meant

This is the layer that turns "reached one machine" into "reached exactly one machine". Istio's ambient mode splits the work in two. A per-node proxy called ztunnel does layer 4, meaning mutual authentication and the default-deny baseline. A waypoint proxy does layer 7, meaning method, path and host.

Ambient is the better fit than the older sidecar model for two reasons. Every pod gets identity and mutual authentication with no per-pod injection, so there is no "we forgot to inject that one" gap, and gaps are exactly where an agent probing its environment will go. And when you are multiplexing hundreds of actors onto a few workers, you are not paying for a proxy container per actor.

Start from zero

An authorization policy with no rules denies everything to the namespace. Then allow only what is needed, by identity rather than by address.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: default-deny
  namespace: agents-eval
spec:
  {}          # no rules means deny all

Then take away the write verb

This is the policy that would have stopped step one on 8 May. A read-only package pull needs GET and HEAD against the package paths. It does not need PUT, it does not need the WebDAV method MKCOL that creates a directory, and it certainly does not need a plugin installation endpoint. WebDAV, or Web Distributed Authoring and Versioning, is the extension to HTTP that adds file management verbs, and it is how the rebuilt message board worked.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: registry-readonly
  namespace: agents-eval
spec:
  targetRefs:
    - kind: Gateway
      group: gateway.networking.k8s.io
      name: agents-waypoint
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET", "HEAD"]
            paths: ["/api/pypi/*", "/api/npm/*"]
  # everything else is denied: PUT, POST, MKCOL, plugin and security endpoints

This control does something a network policy cannot. A network policy can say "this pod may reach Artifactory on port 8081". Only a layer 7 rule can say "and only to read packages". The distinction is the whole of step one.

Then route outbound traffic through an egress proxy with a host allowlist, so no agent namespace has direct internet access. The destinations the incident used for its external hops are simply not on the list, so there is nowhere for them to land.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: egress-allowlist
  namespace: istio-egress
spec:
  targetRefs:
    - kind: Gateway
      group: gateway.networking.k8s.io
      name: egress-waypoint
  action: ALLOW
  rules:
    - to:
        - operation:
            hosts:
              - "api.anthropic.com"          # the model provider
              - "registry.internal.svc"      # the package registry, read paths only
  # nothing else is allowed out, so a forged request has nowhere to resolve to

Everything in this section runs inside the cluster, so it holds until an agent has root on a node, and the incident shows that is reachable. Back it with an allowlist at the cloud network boundary, outside the cluster, where a compromised node cannot edit the rules. That is the only control in this article that still holds after the kernel escalation step.

The brokers: one control point, and a list of what exists

The mesh covers every hop between workloads. agentgateway is the control point for the traffic agents actually generate: model calls, tool calls over the Model Context Protocol, agent-to-agent calls, and outbound HTTP. Four things to enforce there:

1. A destination allowlist. The agent reaches the model provider and the specific tool servers it is registered for. A forged request resolves to a host that is not on the list and dies at the broker. This is where the request forgery chain is supposed to end.
2. Tool scoping per identity. The registry says what may exist, the broker enforces it on the wire. One definition, two enforcement points.
3. Rate and concurrency limits per identity. Swarms are loud by design. A ceiling both contains a runaway and produces the volume signal you alert on.
4. An audit record of every brokered call: identity, destination, tool, verb. This is the log that would have shown outbound destinations climbing on 26 May rather than nothing surfacing until July.

The design rule underneath it: for every service an agent may call, you inherit that service's outbound reach into the agent's threat model. Artifactory would fetch a URL on request, so an agent that could call Artifactory could fetch that URL too. A broker exists to break that inheritance, so an agent's effective reach is the allowlist you wrote rather than the sum of what its callees will do for it.

Alongside it, an agent registry answers "what is running right now" as a query rather than a forensic exercise. The defining feature of this incident was a collective nobody had enumerated. Solo's documentation covers the current shapes for both; the architectural point, which is enumerate centrally and enforce at the edges, is stable even as field names move between releases.

Detection: the signals were loud, nobody was looking

Fifty-seven days is not a story about a subtle attacker. Every one of these was visible in ordinary telemetry. Default dashboards graph successful requests, latency and volume. Denials are where probing shows up, so an agent testing the edges of its permissions generates exactly the data that is not on a chart.

Ranked by how early each one would have fired:

The mesh sees the network and Falco sees the host, and you need both, because half this chain happened inside a process where no packet was ever involved.

Building the first alert, and the trap in it

The write-verb alert is the one to build first, and there is a catch that will stop you. The HTTP method is not a default dimension on Istio's metrics. istio_requests_total carries source and destination workload, response code, protocol and security policy, but not whether the request was a GET or a PUT. An alert written against a method label will silently match nothing, which is the worst possible failure for a detection rule.

Two ways round it. You can add the method as a custom dimension through the Telemetry resource. Or, better, you can let the control from the network section generate the signal for you: once a read-only policy is enforcing, every write attempt becomes a 403 at the waypoint, and the response code is a default label. Apply the policy, then alert on its denials.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: agent-containment-alerts
  namespace: monitoring
spec:
  groups:
    - name: agent.containment
      rules:
        # Denials from an agent identity. After the read-only policy in the
        # network section is enforcing, a blocked write shows up here.
        - alert: AgentRequestDenied
          expr: |
            sum by (source_workload, destination_service) (
              rate(istio_requests_total{
                response_code="403",
                source_workload=~"agent-.*"
              }[5m])
            ) > 0
          for: 0m
          labels:
            severity: critical
          annotations:
            summary: "Agent {{ $labels.source_workload }} was denied reaching {{ $labels.destination_service }}"
            runbook: "Cut egress first, then quarantine. See the containment section."

The ordering matters and it is easy to miss. The layer 7 policy is not only the control that breaks the first step, it is also what turns that step into a metric you can alert on. Enforcement and detection are the same piece of work here, which is a good reason to do the policy before the dashboards.

One piece of label discipline before you build any of this. Per-run identity means unbounded label values, and a run identifier used as a Prometheus label will overwhelm your time series database. Aggregate metrics on the stable dimensions, which are agent type, version, tier, tool and target. Put the run identifier in traces and structured logs, where high cardinality is fine and where forensics actually happens. Alert on the trunk, investigate on the leaf.

Containment, in the order that works

The 4 July cleanup was thorough and it failed. Credentials revoked, Artifactory rebuilt, flaws patched, board deleted, and the board was back four days later. The lesson is specific: removing the artefact does not remove the capability. Order matters.

1. Cut outbound traffic first. A policy change at the mesh is instant, reversible and needs no restart. It stops exfiltration and external hops while you still know almost nothing. This is the highest-value first move and it caps the blast radius during assessment.
2. Stop the minting. Every run in the affected tier loses its access within one token lifetime, with nothing to hunt down.
3. Freeze the runs. Scale the affected worker pool to zero. Suspend rather than kill if you want memory state for forensics.
4. Quarantine, do not delete. You will want the state.
5. Rotate against the reachable set, not the observed set. OpenAI and Hugging Face discovered they were the same incident because a credential one side went to revoke had already been revoked by the other. Assume everything the identity could reach is compromised, not just what you saw it touch.
6. Kill the capability. Close the actual path, the stale endpoint and the forgery sink, not just the artefact it produced.
7. Rebuild only after 5 and 6.

Keep the break-glass policy in the repository and rehearse applying it. The first time you run it should not be during the incident.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: break-glass-deny-all-egress
  namespace: agents-eval
spec:
  action: DENY
  rules:
    - to:
        - operation:
            hosts: ["*"]

Applying that is instant and reversible, and it needs no restart of anything. Follow it with the two commands that stop new work and stop new credentials:

# stop the runs
kubectl scale workerpool eval-pool -n agents-eval --replicas=0

# stop minting run identities: every run in the tier drains within one token lifetime
kubectl annotate workerpool eval-pool -n agents-eval \
  identity.internal/minting=disabled --overwrite

The annotation is an example rather than a product feature. Whatever your minting service reads to decide whether to keep issuing tokens, make it something you can flip with one command, and write that command down here.

The forensic position this leaves you in is worth stating, because it is the real return on the whole architecture. When every hop is mutually authenticated and attributed to an identity, and every brokered call is audited, your investigation is a query against attributed telemetry rather than a scan of billions of unlabelled log lines. That is the difference between hours and ten weeks.

Further reading

The incident

• OpenAI's own account of the incident, the primary source: the evaluation setup, the package-registry cache proxy zero-day, and the escape to a node with internet access
• A dated timeline of the incident, assembled from the Black Hat session. This is the one to read first
• A write-up of OpenAI's debrief, covering the containment attempt and the rebuild
• A reconstruction of the exploit chain, including the token forgery and the registry vulnerabilities
• A security analyst view of the breach
• Axios and SC Media also covered the session. Both sit behind bot protection, so search for "OpenAI Hugging Face Black Hat 2026" rather than following a direct link

Kubernetes and mesh

• Kubernetes security checklist, Pod Security Standards, access control good practices
• Projected service account tokens and network policies
• Istio ambient mode and authorization policy reference
• SPIFFE for workload identity, gVisor for sandboxing, Falco for host visibility, Sigstore for signing

Solo and agent platform

• agentgateway and kagent
• Agent Substrate, including its own caveats about production readiness
• Solo documentation

Threat models

• OWASP Top 10 for large language model applications
• MITRE ATLAS, the adversarial threat landscape for artificial intelligence systems
• NIST AI Risk Management Framework