Virtual keys in agentgateway

What a virtual key actually is

In LiteLLM and Portkey, a virtual key is just a token string you give to a person, a team, or an app. They paste it into whatever makes the request, a curl command, Claude Code, an internal service, and send it on every call. The gateway checks the string, works out who it belongs to, and applies that owner's rules: which models they can use, how fast they can call, how many tokens they get. Then it swaps in your real OpenAI or Anthropic key on the way out, so the person sending requests never sees the real key. One real key sits behind many of these strings, and you can switch any one of them off without touching the real key or anyone else.

So a virtual key is really doing three jobs at once: name who is calling, cap what they can spend, and keep the real provider key hidden. Hold on to those three jobs. They are exactly how agentgateway rebuilds the feature.

The two keys people confuse

Most of the confusion comes from the word "key" doing double duty. When a workshop tells you to "set up a key" for an LLM backend, that is the provider credential. It is not a virtual key. They live in different places and do opposite jobs.

How agentgateway delivers virtual keys

agentgateway builds virtual keys out of three first-class capabilities it already gives you: API-key authentication, token-based rate limiting, and per-key observability. Each is a standard policy you configure on its own, and together they produce the full virtual-key experience: a per-consumer key that identifies the caller, carries its own budget, and resolves to a hidden provider credential.

Composing it from these three pieces is the strength here. The auth, the budgets, and the metrics are independent, so you tune each one to the policy you actually want, layer them with the rest of agentgateway (JWT, prompt guards, model routing) in the same place, and keep everything in plain Kubernetes resources you already manage with GitOps. You get the virtual-key outcome without inheriting one vendor's fixed, all-in-one shape.

The three capabilities map straight onto the three jobs of a virtual key:

How a request flows

Build it: per-user budgets

This is the smallest complete setup: two virtual keys, for Alice and Bob, each with an independent daily budget of 100,000 tokens. Every field below is from the current EnterpriseAgentgatewayPolicy schema.

apiVersion: v1
kind: Secret
metadata:
  name: llm-api-keys
  namespace: agentgateway-system
type: Opaque
stringData:
  alice: |
    {
      "key": "sk-alice-abc123def456",
      "metadata": { "user_id": "alice" }
    }
  bob: |
    {
      "key": "sk-bob-xyz789uvw012",
      "metadata": { "user_id": "bob" }
    }

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: api-key-auth
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
  traffic:
    apiKeyAuthentication:
      mode: Strict
      secretRef:
        name: llm-api-keys

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: daily-token-budget
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
  traffic:
    rateLimit:
      global:
        domain: token-budgets
        backendRef:
          kind: Service
          name: rate-limit-server
          namespace: agentgateway-system
          port: 8081
        descriptors:
        - entries:
          - name: user_id
            expression: 'apiKey.metadata.user_id'
          unit: Tokens

apiVersion: v1
kind: ConfigMap
metadata:
  name: rate-limit-config
  namespace: agentgateway-system
data:
  config.yaml: |
    domain: token-budgets
    descriptors:
    - key: user_id
      rate_limit:
        unit: day
        requests_per_unit: 100000   # 100k tokens/day per user

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: gpt-3.5-turbo
  policies:
    auth:
      secretRef:
        name: openai-secret

An HTTPRoute sending /openai to that backend finishes the wiring. Then it behaves exactly like a virtual-key system: Alice's key works until her 100,000 tokens are gone and she starts getting 429s, while Bob keeps going on his own untouched budget.

# Alice's virtual key
curl "$INGRESS_GW_ADDRESS/openai" \
  -H "Authorization: Bearer sk-alice-abc123def456" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello!"}]}'

# Once her budget is spent:
# HTTP/1.1 429 Too Many Requests
# x-ratelimit-limit: 100000
# x-ratelimit-remaining: 0

Tiered budgets by user type

Because identity is just metadata on the key, you scale budgets by adding a field. Tag each key with a tier, then add it as a second descriptor entry. Free, standard, and premium users now draw from different daily limits.

# In the Secret
alice: |
  { "key": "sk-alice-abc123def456",
    "metadata": { "user_id": "alice", "tier": "premium" } }

# In the rateLimit descriptor
descriptors:
- entries:
  - name: tier
    expression: 'apiKey.metadata.tier'
  - name: user_id
    expression: 'apiKey.metadata.user_id'
  unit: Tokens

domain: token-budgets
descriptors:
- key: tier
  value: "free"
  descriptors:
  - key: user_id
    rate_limit: { unit: day, requests_per_unit: 10000 }
- key: tier
  value: "premium"
  descriptors:
  - key: user_id
    rate_limit: { unit: day, requests_per_unit: 500000 }

Multi-tenant virtual keys

The same move scopes keys to a tenant as well as a user. Add a tenant_id to the key metadata and lead the descriptor with it, so every user's budget is nested under their tenant.

# In the rateLimit descriptor
descriptors:
- entries:
  - name: tenant_id
    expression: 'apiKey.metadata.tenant_id'
  - name: user_id
    expression: 'apiKey.metadata.user_id'
  unit: Tokens

Shorter budget windows

The refresh interval lives on the rate-limit server, not the policy. Drop unit from day to hour (or minute, second) for tighter control, so a runaway key can only burn its allowance for an hour before it refills rather than a whole day.

# In the rate-limit-config ConfigMap
domain: token-budgets
descriptors:
- key: user_id
  rate_limit:
    unit: hour
    requests_per_unit: 10000   # 10k tokens/hour per user

Track per-key spend

The third job, knowing what each key actually costs, comes from the token-usage metric, broken down by the same user_id. agentgateway records input and output tokens per request, so you can total a user's daily consumption and turn it into money with your provider's pricing.

# Total tokens per user over the last 24h
sum by (user_id) (
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h]) +
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])
)

# Percentage of a 100k daily budget used, per user
(sum by (user_id) (
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h]) +
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])
) / 100000) * 100

# Cost per user (example: $0.50 / 1M input, $1.50 / 1M output)
sum by (user_id) (
  ((rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h])  / 1e6) * 0.50) +
  ((rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h]) / 1e6) * 1.50)
)

Checklist