Agent-to-Agent in kagent: an SRE that delegates database incidents to a DBA

{
  "name": "dba_agent",
  "description": "Database SRE specialist. Diagnoses Postgres/database workload problems ...",
  "url": "http://kagent-controller.kagent.svc.cluster.local:8083/api/a2a/kagent/dba-agent/",
  "capabilities": { "streaming": true, "pushNotifications": false, "stateTransitionHistory": true },
  "defaultInputModes": ["text"],
  "defaultOutputModes": ["text"],
  "skills": [
    {
      "id": "diagnose-db",
      "name": "Diagnose a database incident",
      "description": "Find the root cause of a database workload failure and propose the fix",
      "tags": ["database", "postgres", "sre"],
      "examples": ["The orders database pod is crashlooping, why?", "Postgres won't start after the last deploy"]
    }
  ],
  "preferredTransport": "JSONRPC"
}

POST / HTTP/1.1
Host: dba-agent.kagent:8080
User-Agent: python-httpx/0.28.1
x-kagent-source: agent
x-user-id: ca0c9432-6f36-44cc-9fd5-c66048cdfc37

{
  "jsonrpc": "2.0",
  "id": "cff5fae8-7be6-4373-a2c3-1af848a38b62",
  "method": "message/send",
  "params": {
    "configuration": { "acceptedOutputModes": [], "blocking": true },
    "message": {
      "role": "user",
      "kind": "message",
      "contextId": "7d068fc6-3a67-4586-bf36-58c49e43b272",
      "messageId": "4d1f7743-ad78-4b74-aeaf-aed4a8973e0e",
      "parts": [{ "kind": "text", "text":
        "The orders-db pod (postgres:16-alpine) in the orders namespace is in
         CrashLoopBackOff with 30 restarts. The container exits with code 1 ...
         Environment: POSTGRES_DB=orders ... Please diagnose why the PostgreSQL
         container is failing to start and what should be fixed." }]
    }
  }
}

{
  "jsonrpc": "2.0",
  "id": "cff5fae8-...",
  "result": {
    "kind": "task",
    "id": "08508326-...",
    "contextId": "135882bd-...",
    "status": { "state": "completed" },
    "history": [ /* 9 messages: the DBA's describe/logs/events steps */ ],
    "artifacts": [ { "parts": [ { "kind": "text", "text": "...diagnosis below..." } ] } ]
  }
}

## Root Cause

The `orders-db` pod is crashlooping because the `POSTGRES_PASSWORD` environment
variable is not set. The container is configured with only POSTGRES_DB=orders, but
the postgres:16-alpine image requires either POSTGRES_PASSWORD for the superuser,
or POSTGRES_HOST_AUTH_METHOD=trust (not recommended). Without it, initialization
fails with exit code 1 and the pod crashloops.

## Exact Fix

1. Create a Secret with the superuser password:
   kubectl create secret generic orders-db-secret \
     --from-literal=password='<strong-password>' -n orders
2. Add POSTGRES_PASSWORD to the deployment from that Secret (secretKeyRef).
3. Verify:
   kubectl rollout status deployment/orders-db -n orders
   kubectl get pod -n orders -l app=orders-db

kubectl -n kagent port-forward svc/kagent-controller 8083:8083 &

# discover: the agent card
curl localhost:8083/api/a2a/kagent/sre-orchestrator/.well-known/agent.json

# send a task (note the trailing slash on the agent path)
curl -X POST "http://localhost:8083/api/a2a/kagent/sre-orchestrator/" \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send",
       "params":{"message":{"role":"user",
         "parts":[{"kind":"text","text":"the orders database won'\''t start"}],
         "messageId":"m1"}}}'

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-orchestrator
  namespace: kagent
spec:
  type: Declarative
  description: On-call SRE orchestrator. Triages cluster incidents and delegates
    database-specific diagnosis to the dba-agent.
  declarative:
    modelConfig: default-model-config
    systemMessage: |
      You are the on-call SRE. Triage incidents by inspecting Kubernetes
      resources and events. When the problem is database-related DELEGATE the
      diagnosis to the dba-agent and incorporate its findings. Produce a short
      incident summary: what is broken, the root cause, the recommended
      remediation. Note which finding came from the DBA specialist.
    tools:
      - type: McpServer                     # its own k8s read tools
        mcpServer:
          kind: RemoteMCPServer
          name: kagent-tool-server
          toolNames: [k8s_get_resources, k8s_get_events, k8s_describe_resource]
      - type: Agent                         # <-- A2A delegation
        agent:
          name: dba-agent

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: dba-agent
  namespace: kagent
spec:
  type: Declarative
  description: Database SRE specialist. Diagnoses Postgres/database workload
    problems from Kubernetes state.
  declarative:
    modelConfig: default-model-config
    systemMessage: |
      You are a database reliability specialist for the platform's Postgres
      databases. Given a symptom, inspect the relevant Deployment and pods
      (describe, logs, events), then explain the root cause in plain language and
      give the exact remediation. You diagnose and recommend; you do not apply
      destructive changes yourself.
    tools:
      - type: McpServer
        mcpServer:
          kind: RemoteMCPServer
          name: kagent-tool-server
          toolNames: [k8s_get_resources, k8s_describe_resource, k8s_get_pod_logs, k8s_get_events]
    a2aConfig:                              # <-- becomes the agent card
      skills:
        - id: diagnose-db
          name: Diagnose a database incident
          description: Find the root cause of a database workload failure and propose the fix
          inputModes: [text]
          outputModes: [text]
          tags: [database, postgres, sre]
          examples:
            - "The orders database pod is crashlooping, why?"
            - "Postgres won't start after the last deploy"

apiVersion: v1
kind: Namespace
metadata: { name: orders, labels: { purpose: a2a-demo } }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-db
  namespace: orders
  annotations: { incident/summary: "orders Postgres will not start after a deploy" }
spec:
  replicas: 1
  selector: { matchLabels: { app: orders-db } }
  template:
    metadata: { labels: { app: orders-db } }
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          # BROKEN ON PURPOSE: no POSTGRES_PASSWORD and no trust auth method.
          env:
            - { name: POSTGRES_DB, value: orders }
          ports: [{ containerPort: 5432 }]
          resources:
            requests: { cpu: 10m, memory: 32Mi }
            limits:   { cpu: 250m, memory: 128Mi }
---
apiVersion: v1
kind: Service
metadata: { name: orders-db, namespace: orders }
spec:
  selector: { app: orders-db }
  ports: [{ port: 5432, targetPort: 5432 }]

# Alice's group may call the orchestrator (matched on the token's groups claim).
apiVersion: policy.kagent-enterprise.solo.io/v1alpha1
kind: AccessPolicy
metadata: { name: allow-fieldfte-orchestrator, namespace: kagent }
spec:
  action: ALLOW
  from:
    subjects:
      - kind: UserGroup
        userGroup:
          claimName: groups
          claimValue: field-fte
          issuer: http://keycloak.keycloak.svc.cluster.local/realms/solo
  targetRef: { kind: Agent, name: sre-orchestrator }
---
# The orchestrator (acting agent) may call the dba-agent.
apiVersion: policy.kagent-enterprise.solo.io/v1alpha1
kind: AccessPolicy
metadata: { name: allow-orchestrator-to-dba, namespace: kagent }
spec:
  action: ALLOW
  from:
    subjects: [{ kind: Agent, name: sre-orchestrator, namespace: kagent }]
  targetRef: { kind: Agent, name: dba-agent }
---
# The end user may NOT call the dba-agent directly.
apiVersion: policy.kagent-enterprise.solo.io/v1alpha1
kind: AccessPolicy
metadata: { name: deny-user-direct-dba, namespace: kagent }
spec:
  action: DENY
  from:
    subjects:
      - kind: UserGroup
        userGroup: { claimName: groups, claimValue: field-fte, issuer: http://keycloak.keycloak.svc.cluster.local/realms/solo }
  targetRef: { kind: Agent, name: dba-agent }

# helm upgrade --install kagent oci://.../kagent-enterprise  (key values)
providers:
  default: anthropic
  anthropic: { apiKey: $ANTHROPIC_API_KEY }
oidc:
  issuer: http://keycloak.keycloak.svc.cluster.local/realms/solo
  clientId: kagent
  skipOBO: false                 # OBO on: exchange, don't just forward
kagent-tools: { enabled: true }  # bundled k8s MCP tool server
rbac:
  roleMapping:
    # map the Keycloak groups claim (lowercase!) to kagent roles
    roleMapper: 'claims.groups.transformList(i, v, v in rolesMap, rolesMap[v])'
    roleMappings:
      field-fte: global.Admin
      field-trial: global.Reader
      field-admin: global.Admin

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: keycloak, namespace: keycloak }
spec:
  serviceName: keycloak
  replicas: 1
  selector: { matchLabels: { app: keycloak } }
  template:
    metadata: { labels: { app: keycloak } }
    spec:
      containers:
        - name: keycloak
          image: quay.io/keycloak/keycloak:26.3
          args: [start-dev, --import-realm]
          env:
            # Pin the issuer so the token `iss` matches the controller's oidc.issuer
            # whether minted via port-forward or in-cluster.
            - { name: KC_HOSTNAME, value: "http://keycloak.keycloak.svc.cluster.local" }
            - { name: KC_HOSTNAME_STRICT, value: "false" }
            - { name: KC_HOSTNAME_BACKCHANNEL_DYNAMIC, value: "true" }
          volumeMounts:
            - { name: realm-import, mountPath: /opt/keycloak/data/import, readOnly: true }
      volumes:
        - name: realm-import
          configMap: { name: keycloak-realm-import }
# realm `solo`: users alice/field-fte, bob/field-trial, carol/field-admin;
# public client `kagent` with a groups mapper and an audience=kagent mapper.