Two axes: Q1–Q6 identity (who can call what, how identity flows through the chain) and Q7+ runtime safety (what the gateway should still be doing once the caller is legitimate). Each one starts with what we see in the field, then drops into the AGW pattern that implements it. Where we already have a deep-dive on a topic, a "Read the deep-dive →" link points at it so this page stays a navigable overview rather than a wall. The closing section maps every question onto the OWASP, NIST AI RMF and MITRE ATLAS categories GRC teams care about.
Stack at a glance — what's upstream vs Solo Enterprise
Throughout the sections you'll see two pills next to CRD names and features — OSS means upstream agentgateway 2.1 ships it today, ENT means it requires Solo Enterprise for agentgateway, and ENT ISTIO means it requires Solo Enterprise for Istio (mesh-side capabilities like Ambient cross-cluster peering). Here's the high-level split.
Upstream / OSS — use today
AgentgatewayPolicy(agentgateway.dev)AgentgatewayBackend·AgentgatewayParameters- MCP-protocol-aware routing &
mcp.tool.nameCEL authz promptGuard.request/responsewith built-in detectors + custom regex + webhook- Guardrail Webhook API (Beta in 2.1.x)
- A2A app-protocol backend (
kgateway.dev/a2a) - Upstream Istio (sidecar or Ambient, single-cluster)
Solo Enterprise for agentgateway
EnterpriseAgentgatewayPolicy+EnterpriseAgentgatewayParametersAuthConfig(extauth.solo.io) — OAuth2 / OIDC ext-authzRateLimitConfig(ratelimit.solo.io) — CEL + global rate limitingtraffic.jwtAuthentication.mode: Stricttraffic.transformation(CEL-based request/response edits)- AGW STS — RFC 8693 token-exchange minting with nested
actclaim - Semantic cache · model failover
- Solo Agent Registry — agent/MCP/tool catalog as the source of truth
Solo Enterprise for Istio
- Ambient cross-cluster peering — east-west GW + auto-rewrite of endpoints
topology.istio.io/networkclassification across clusters- Per-cluster intermediate CAs sharing a single trust domain
- Gloo mgmt-server xDS relay (multi-cluster federation control plane)
- Solo Mesh governance overlays —
Workspace,WorkspaceSettings,AccessPolicy
Every Q section below uses CRDs from one or more of these columns. The pills next to each CRD or feature mention tell you which.
Q1 How do we handle authn & authz across MCP Gateway, MCP Server and tools — and which standards fit?
There isn't one auth boundary — there are three hops and each one is handled differently. The often-forgotten fourth hop is the tool's call to its own backing API, which varies too much to standardise here.
| Hop | What we see | Recommendation |
|---|---|---|
| A · Client → MCP Gateway | Everyone's landed on OAuth 2.0 + OIDC, strict JWT validation at the gateway, audience + issuer pinning. API keys still creep in. | OAuth 2.0 + OIDC, validate the JWT at the gateway. Pin aud and iss. No long-lived API keys. |
| B · MCP Gateway → MCP Server | Three patterns in the wild, in increasing maturity: token pass-through (kills audit + OBO), Istio workload identity + mTLS, RFC 8693 token exchange with a nested act claim. |
Token exchange (RFC 8693) with a nested act claim. mTLS underneath via the mesh. |
| C · MCP Server → tools / backends | Tool-call authz often lives inside the MCP server, which means every server re-implements it. AGW is MCP-protocol aware, so mcp.tool.name is visible to policy at the gateway. |
Move "can this caller invoke this tool" out of the server and into the gateway. Server stays thin. |
Standards worth using
- OAuth 2.0 + OIDC at the edge
- RFC 8693 token exchange with nested
actfor every gateway-to-server / agent-to-agent hop - mTLS / SPIFFE under it all, courtesy of the mesh
- RFC 9396 Rich Authorization Requests (
authorization_details) when scope sprawl starts hurting
Avoid
- Token pass-through (no audit boundary, no OBO)
- Long-lived service tokens that never rotate
- Agent-supplied identity headers (anything self-asserted)
- Treating IdP scopes as the only authz check (see Q2)
AuthConfig plugin chain — including JWT verification,
OAuth2 introspection, OPA Rego and claimsToHeaders —
are covered in detail at
Solo external auth service.
OIDC and on-behalf-of in Ambient is at
JWT, OIDC and on-behalf-of in Istio Ambient.
Q2 At what level should permissions be enforced — tool, action, resource or context?
Four levels, in order of maturity. Most teams start at the top of the ladder and move through them as they deploy and get more familiar with the surface. By default we recommend tool-level allowlist + a coarse scope check — explicit, default-deny on anything new.
| Level | Question it answers | Where it lives |
|---|---|---|
| Route | Can this caller reach this MCP at all? (e.g. only callers in the finance role can reach the finance MCP) | Gateway — CEL on jwt.groups |
| Tool | Can this caller invoke this specific tool? (can call read tools, not write tools) | Gateway — CEL on mcp.tool.name |
| Action / argument | Can this caller invoke this tool with these arguments? (only on their own accounts, only under a $ limit) | Tool implementation — gateway doesn't see tool args today |
| Context | Is this allowed right now? (valid OBO chain, request-time checks) | Gateway — CEL on jwt.act.* and headers |
Sample policy — a GitHub MCP locked to read-only operations,
layering a coarse scope envelope with an explicit tool list
(AgentgatewayPolicy OSS):
backend:
mcp:
authorization:
action: Allow
policy:
matchExpressions:
- >-
jwt.scope.contains("github-readonly") &&
mcp.tool.name in [
"get_file_contents","search_repositories","search_code","search_issues",
"search_users","list_commits","get_issue","list_issues",
"get_pull_request","list_pull_requests","list_branches",
"list_pull_request_files"
]
Per-user is just another clause:
jwt.sub == "alice" && mcp.tool.name == "get_me"
mcp, Atlassian's
read:jira-work is finer but still huge.
Scope-based authz at the MCP server boundary doesn't actually
constrain behaviour in practice. Enforce at the
gateway, against mcp.tool.name. Use
OAuth Rich Authorization Requests (RFC 9396,
authorization_details) to avoid scope sprawl —
RAR carries intent in the token, CEL enforces it at call
time. They compose well.
AgentgatewayPolicy with mcp.tool.name
enforcement — is walked through at
TrustUsBank — agentic DORA demo.
Q3 Which access-control model fits — RBAC, ABAC or TBAC?
Don't pick one — layer them. Each model answers a different question and they compose into a single CEL expression at the gateway.
| Layer | Question it answers | Carrier |
|---|---|---|
| A · RBAC (user role) | Which roles can hit which agents / MCPs? | IdP groups → jwt.groups → CEL |
| B · RBAC (capability classes) | Which tool can this caller invoke? | mcp.tool.name in [...] + jwt.scope |
| C · ABAC | Is this OBO chain / context valid right now? | CEL on jwt.act.* and headers |
Layered together — a single CEL expression that demands all three:
jwt.groups.exists(g, g == "investment-advisors") &&
jwt.scope.contains("bank-agents") &&
mcp.tool.name in ["get_user_financial_profile","analyze_portfolio","project_retirement"]
risk_tier of the tool being called. Encoded as CEL
against jwt.act.*, headers and request context at
the gateway. Auditors care about the chain and the context, not
just the tool list.
If you carry per-tenant or per-organisational-unit labels on
your IdP groups (a common ask from banks),
jwt.groups is the natural place to carry them too
— the same claim drives RBAC and per-tenant routing.
claimsToHeaders on a kgateway
AuthConfig ENT — so downstream routing and rate
limiting can key off x-tenant-id without
re-parsing the JWT — is covered at
JWT claims to HTTP headers.
Q4 How should we group and govern tools — by domain, function, risk or data sensitivity?
Group tools by risk, not by which MCP they happen to live in. Risk drives how much enforcement each tool needs — and it's the axis regulators ask about.
Q5 How do AI Gateway, MCP Gateway, MCP Server, tools and agents associate — and how do they discover one another?
One of the practical wins with AGW: the same product handles LLM, MCP, and A2A traffic. You don't run a separate LLM Gateway, MCP Gateway, and agent gateway as three products. There are two deployment patterns, and the waypoint one is the more interesting story for east-west.
| Pattern | When | How agents see it |
|---|---|---|
| A · Explicit gateway | North-south, and east-west if there's no mesh. One AGW deployment, multiple Gateway resources. Segregate by listener (multiple Gateway objects) or by route. | Agents call the gateway hostname explicitly. |
| B · AGW as an ambient waypoint | East-west, transparent. Agents and MCPs already on an Istio Ambient mesh. AGW deploys as the waypoint; mesh routes through it. | Agents don't know AGW is in the path — they call company-mcp:3000/mcp directly. Mesh handles the routing, AGW enforces the policy. |
Service-to-service traffic gets the same MCP authz, OBO chain enforcement, prompt guards, rate limits and audit either way — just attached as a waypoint instead of an explicit hop. See the Solo blog From Service Mesh to Agentic Mesh for the full pattern.
Principles regardless of pattern:
- MCP servers behind AGW, never exposed directly. Workload identity from gateway → server.
- Agents are clients, not infra. Each agent has its own OAuth client, scopes, quota.
Where to attach policies — Gateway, HTTPRoute, AgentgatewayBackend in targetRefs:
| Attach point | Best for |
|---|---|
| Gateway-level | Blanket apply — the cleanest default. |
| HTTPRoute-level | When some endpoints must stay unauthenticated (OIDC discovery, token endpoint). |
| AgentgatewayBackend-level | Where MCP tool-authz lives — closest to the resource. |
Discovery:
- Workload discovery — Kubernetes Service + xDS in-cluster OSS; Solo Istio for cross-cluster ENT ISTIO.
- Agent / tool catalog — Solo Agent Registry ENT (which agents/MCPs exist, what tools they expose, what's in which environment).
Logging — split into two streams, run them separately:
A · Operational
- OTel via OTLP
- Distributed-tracing context propagated at every hop
- A user → Butler → Coach → MCP call shows up as one trace
- ClickHouse at scale as the sink
B · Audit
- Separate access-log policy on the gateway
- CEL-extracted attributes for what auditors ask about:
jwt.sub(caller),jwt.act.sub/jwt.act.act.sub(agent chain),mcp.tool.name(tool invoked), decision, policy version, timestamp - Add a
reason_codelabel (entitlement_denied,scope_missing,quarantined) — a stable label, not the data that triggered the deny risk_tierfrom the Q4 catalog as a span attribute, plus budget-consumption counters (turns,tokens_in,tokens_out) — pairs naturally with Q9- Never log the payload, never log a successful tool's return body — hashes + labels only
Q6 When agents invoke other agents, how do we propagate identity and stop privilege escalation?
When an agent calls another agent, identity and context have
to narrow as they move down the chain — never
widen. The RFC 8693 token exchange pattern with a nested
act claim is the answer at every hop.
The AGW STS ENT mints a fresh, downscoped, audience-pinned token
at each hop. The previous caller becomes a nested
act claim, so the full chain is recoverable from
the token itself:
{
"iss": "https://agw-sts.example.com",
"sub": "user-12345",
"aud": "mcp-finance-advices",
"scope": "bank-agents",
"act": {
"sub": "agent-coach",
"act": { "sub": "agent-butler" }
}
}
At sensitive resources, assert the full chain — not just the last actor:
jwt.iss == "https://agw-sts.example.com" &&
jwt.act.sub == "agent-coach" &&
jwt.act.act.sub == "agent-butler"
A single-actor check would miss "skip Butler, user goes straight to Coach". The chain check catches it.
act claim nesting
at each hop) is at
JWT token exchange — OIDC IdP + agentgateway STS walkthrough.
Q7 How do we enforce strict schemas on every MCP call and A2A hop?
Even a perfectly authorized caller can ship a malformed JSON-RPC envelope, a JWT with the wrong audience, or a tool payload twice the size you expect. The gateway is the right place to drop those before they reach the tool process.
| Envelope | What AGW enforces today | How |
|---|---|---|
| MCP JSON-RPC | Malformed JSON-RPC dropped at the gateway, never reaches the server. | Native — AGW is MCP-protocol-aware. |
| JWT | Strict-mode validation, JWKS pinned, aud + iss matched. |
traffic.jwtAuthentication.mode: Strict ENT |
| A2A | A2A request shape enforced before the agent runtime. | appProtocol: kgateway.dev/a2a OSS on the backend. |
JWT envelope in Strict mode, JWKS pulled from
the IdP — the snippet that does the actual hard work:
spec:
traffic:
jwtAuthentication:
mode: Strict
providers:
- issuer: "https://idp.example.com"
jwks:
remote:
jwksPath: "/protocol/openid-connect/certs"
backendRef:
name: identity-provider
namespace: auth-system
kind: Service
port: 8080
amount field must be a
positive integer below 100". For that, drop in the
Custom Guardrails Webhook API — shipping
today in agentgateway 2.1.x and 2.3.x, currently labelled
Beta. The webhook receives the normalized
request (and response) over HTTP and returns one of three
actions: Pass, Mask or
Reject. The pattern: have it load each tool's
JSON Schema from the registry and reject pre-LLM — same
source of truth as Q4's tool catalog.
Q8 How do we keep tool output from poisoning the next prompt?
An MCP tool that returns a record from a database is returning text the LLM is about to read and act on. If that text carries a prompt-injection payload, an SSN, or a chunk of base64-encoded data the next agent will be asked to "forward" — that becomes the gateway's problem, not a "be careful, agent" instruction inside the system prompt.
Response mask with built-in detectors OSS — the common case:
spec:
backend:
ai:
promptGuard:
response:
- regex:
action: Mask
builtins:
- "CreditCard"
- "Ssn"
- "Email"
- "PhoneNumber"
- "CaSin"
Where built-ins aren't enough — language-specific PII, base64 / entropy heuristics, model-driven moderation — swap the regex for a webhook:
spec:
backend:
ai:
promptGuard:
request:
- webhook:
backendRef:
kind: Service
name: ai-guardrail-webhook
port: 8000
response:
- webhook:
backendRef:
kind: Service
name: ai-guardrail-webhook
port: 8000
The webhook returns one of three actions per request and
per response: Pass, Mask or
Reject. Run different policies on different
surfaces — user input is not the same threat as MCP tool
output is not the same threat as agent-to-agent.
Worth doing
promptGuard.responsewith built-in masks on every MCP route returning customer data- Custom guardrail webhook for everything the built-ins don't cover
- Different policies per surface — user→agent, agent→MCP, agent→agent
- System instructions in fixed templates only — never free-concatenate user-supplied text into them
Avoid
- Trusting the LLM to recognise prompt injection in tool output it's already loaded
- One giant prompt-guard policy that tries to cover every surface with the same rules
- Block-only when
Maskwould preserve the useful parts of the response
Q9 How do we stop a runaway agent loop — turns, depth, repetition?
An agent that calls another agent that calls the first agent back, or an LLM that keeps invoking the same tool with slightly different arguments hoping for a different answer — these don't trip auth, schema or output gating. They eat budget and they're how an agent does denial-of-service to itself.
| Axis | Native in AGW today | Needs a stateful component |
|---|---|---|
| Token / cost budget | RateLimitConfig ENT per-user tiered (minute / hour / day); token-budget metrics via agentgateway_gen_ai_client_token_usage |
— |
| Max turns · max tool calls · chaining depth | — | Per-trace counter (Redis-backed) called on every MCP / A2A hop via the Guardrail Webhook API (Beta). |
| Repetition (same tool + same args; A2A ping-pong) | — | Same session store. Trace ID is already propagated at every hop. |
Per-user tiered rate limit (request-based, real
RateLimitConfig ENT):
apiVersion: ratelimit.solo.io/v1alpha1
kind: RateLimitConfig
metadata:
name: per-user-tiered-rate-limit
namespace: agentgateway-system
spec:
raw:
descriptors:
- key: X-User-ID
descriptors:
- key: per-minute
rateLimit: { requestsPerUnit: 20, unit: MINUTE }
- key: per-hour
rateLimit: { requestsPerUnit: 500, unit: HOUR }
- key: per-day
rateLimit: { requestsPerUnit: 5000, unit: DAY }
Q10 When should the gateway pause for a human, and how is that approval bound to the request?
Human-in-the-loop is the right answer for irreversible,
high-impact actions — Tier 3 in Q4. The
shipped answer today lives at the agent runtime, not
at the gateway: kagent 0.8.0 added a requireApproval
field that pauses tool execution and surfaces an Approve / Reject
prompt in the kagent dashboard before the call goes out.
The shipped flow — kagent requireApproval OSS:
- On the agent's MCP tool-server stanza, list the high-impact tools under
requireApproval(must be a subset oftoolNames). - When the model decides to call one of them, execution pauses before the tool fires.
- The kagent dashboard renders the pending call — tool name, arguments — and shows Approve / Reject.
- Approve → the tool executes. Reject → the rejection reason is fed back to the model so it can adapt or pick a different path.
- Same pattern works through an orchestrator that delegates to a subagent: the approval bubbles up through the chain to the dashboard.
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
name: k8s-agent
namespace: infra
spec:
type: Declarative
declarative:
systemMessage: |
You manage Kubernetes resources. Before deleting or applying,
the user will be asked to confirm. Explain what you plan to do.
modelConfig: default-model
tools:
- type: McpServer
mcpServer:
name: k8s-tool-server
kind: RemoteMCPServer
apiGroup: kagent.dev
toolNames:
- k8s_get_resources
- k8s_describe_resource
- k8s_delete_resource
- k8s_apply_manifest
requireApproval:
- k8s_delete_resource
- k8s_apply_manifest
There's also a built-in ask_user tool every agent
can call to ask a structured question (single-select,
multi-select, free-text). That one is model-initiated
("I need clarification"). requireApproval is
tool-initiated ("this tool needs sign-off before it
runs"). Use them together.
(tool, args, caller): the approval UI mints a
short-lived JWT whose aud claim is
SHA256(tool || args || principal || trace_id),
and a Rego rule at the gateway recomputes the hash and accepts
only if it matches. This is not a shipped product
feature — it's a pattern you build on top of OIDC + AGW STS
ENT + Rego body inspection.
The Rego patterns that make it possible (subject lookup,
audience check, deny-list, layered OAuth + OPA) are walked
through at Solo external auth
service.
Q11 How do we kill a tool, route or session in seconds without a redeploy?
Real-world incident response needs a kill switch that flips a single tool, route or session off in seconds — no redeploy, no rebuild, no waiting for a Helm release. The audit story matters too: what was killed, when, by whom.
The pattern — flip the entitlements allowlist by kubectl apply:
- Day-1 policy: every tool is gated by the Q2 tool allowlist (
mcp.tool.name in [...]OSS). - To quarantine a tool, remove it from the allowlist (or add a deny clause) and
kubectl apply. Propagation is ~2 seconds on AGW. - To release, re-add and re-apply — same latency.
- Audit trail: git log of the policy YAML + OTel deny spans on every blocked request after the change.
The same allowlist from Q2, with one tool removed (note the git-friendly comment — your audit trail starts in the diff):
spec:
backend:
mcp:
authorization:
action: Allow
policy:
matchExpressions:
- >-
mcp.tool.name in [
"get_file_contents","search_repositories","list_issues"
# "bulk_export" removed 2026-05-21 — quarantined
]
make quarantine TOOL=... command, a
kagent-UI button) is a one-day build. Session-level
quarantine reuses the same session store as Q9.
AgentgatewayPolicy shape are at
TrustUsBank — agentic DORA demo.
Where these questions land in industry frameworks
GRC teams want evidence in the language of the frameworks they've already adopted. Each question above tackles a category that OWASP (LLM Top 10 + the Agentic Security Initiative), the NIST AI Risk Management Framework, and MITRE ATLAS (the adversarial-ML catalog) already track. The mapping isn't strict 1:1 — frameworks overlap and frame things differently — but the high-level coverage looks like this:
| Question | OWASP (LLM Top 10 + Agentic AI) | NIST AI RMF (function) | MITRE ATLAS (tactic) |
|---|---|---|---|
| Q1–Q3 authn / authz | Excessive Agency · Identity Spoofing | Govern · Manage | Initial Access · Defense Evasion |
| Q4 tool grouping by risk | Tool Misuse | Manage · Map | Discovery |
| Q5 observability & audit | Sensitive Information Disclosure · Repudiation | Measure | cross-cutting |
Q6 chain identity (act claim) |
Identity Spoofing · Authority Compromise | Govern | Initial Access |
| Q7 strict schemas | Improper Output Handling · Tool Definition Poisoning | Manage | Defense Evasion |
| Q8 output gating + injection-strip | Prompt Injection · Sensitive Information Disclosure | Manage | ML Attack Staging · Exfiltration |
| Q9 runaway containment | Unbounded Consumption · Resource Overload | Manage | Impact |
| Q10 HITL approvals | Excessive Agency · Human Manipulation | Govern | Impact |
| Q11 quarantine / kill switch | Misaligned Behaviors | Manage | cross-cutting |
Whichever framework a customer's GRC team already reports against, the answers above slot into categories they're filing evidence for. The deep-dive links in each section give you the YAML that is that evidence.
Thanks again to Ram Vennam for the source material and for the field perspective behind every one of these answers.