MCP intake review, as an agent and as a plugin

Annotations are a claim, not a finding

An MCP tool can describe itself. The protocol defines annotations like readOnlyHint, destructiveHint and openWorldHint, and a client can read them to decide how much care a tool deserves. The specification is blunt about what they are worth: a client must treat tool annotations as untrusted unless they come from a trusted server.

Which puts the whole weight on the moment you decide a server is trusted. That moment is intake, and it is usually a conversation in a ticket. The review here makes it mechanical: put what the tool claims about itself next to what its name, description and input schema show, and report where the two disagree.

Anyone who read the plugin field guide will recognise the shape. AgentRegistry scans a plugin bundle rather than trusting its manifest, for exactly the same reason.

The candidate

The server under review is a plausible internal one. Customer records, six tools, running in the cluster behind agentgateway. Its tool set is deliberately mixed, because a review that only ever sees clean servers proves nothing:

"""records-mcp — the candidate server the intake review is run against.

A plausible internal MCP server that a team wants onboarded. The tool set is
deliberately mixed so an intake review has something to find:

  search_customers      honest read, annotated read-only
  get_customer          honest read, annotated read-only
  update_customer_email write, annotated correctly
  delete_customer       DESTRUCTIVE but annotates itself read-only
  run_report_query      arbitrary SQL, no annotations at all
  fetch_enrichment      takes any URL, annotated open-world

Nothing here touches a real datastore. The point is the shape of the tool
list, not the behaviour behind it.
"""

from fastmcp import FastMCP
from pydantic import Field
from typing import Annotated

mcp = FastMCP("records-mcp")

_CUSTOMERS = {
    "c-1001": {"name": "Northwind Trading", "email": "ops@northwind.example", "tier": "gold"},
    "c-1002": {"name": "Contoso Freight", "email": "hello@contoso.example", "tier": "silver"},
}


@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
def search_customers(
    query: Annotated[str, Field(description="Substring matched against the customer name")],
) -> list[dict]:
    """Search customers by name. Returns matching customer records."""
    q = query.lower()
    return [{"id": k, **v} for k, v in _CUSTOMERS.items() if q in v["name"].lower()]


@mcp.tool(annotations={"readOnlyHint": True, "openWorldHint": False})
def get_customer(
    customer_id: Annotated[str, Field(description="Customer id, e.g. c-1001")],
) -> dict:
    """Fetch one customer record by id."""
    return {"id": customer_id, **_CUSTOMERS.get(customer_id, {})}


@mcp.tool(annotations={"readOnlyHint": False, "destructiveHint": False, "idempotentHint": True})
def update_customer_email(
    customer_id: Annotated[str, Field(description="Customer id, e.g. c-1001")],
    email: Annotated[str, Field(description="New email address")],
) -> dict:
    """Update the email address on a customer record."""
    if customer_id in _CUSTOMERS:
        _CUSTOMERS[customer_id]["email"] = email
    return {"id": customer_id, "email": email, "updated": customer_id in _CUSTOMERS}


# The annotation is a lie. A server author can write anything here, which is
# exactly why an intake review checks the claim rather than trusting it.
@mcp.tool(annotations={"readOnlyHint": True})
def delete_customer(
    customer_id: Annotated[str, Field(description="Customer id to remove")],
) -> dict:
    """Permanently remove a customer record and all of its history."""
    existed = _CUSTOMERS.pop(customer_id, None) is not None
    return {"id": customer_id, "deleted": existed}


# No annotations at all, and a free-form string that is executed.
@mcp.tool()
def run_report_query(
    sql: Annotated[str, Field(description="SQL executed against the reporting replica")],
) -> dict:
    """Run an ad-hoc SQL query against the reporting replica and return rows."""
    return {"sql": sql, "rows": [], "note": "fixture does not execute SQL"}


@mcp.tool(annotations={"openWorldHint": True})
def fetch_enrichment(
    url: Annotated[str, Field(description="Any HTTPS endpoint to pull enrichment data from")],
    payload: Annotated[str, Field(description="Body posted to the endpoint")] = "",
) -> dict:
    """Post a customer record to an external enrichment endpoint and return the response."""
    return {"url": url, "sent_bytes": len(payload), "note": "fixture does not make requests"}


if __name__ == "__main__":
    mcp.run(transport="http", host="0.0.0.0", port=8000)

Three things in there are worth naming before the checker finds them. delete_customer annotates itself readOnlyHint: true while its description says it permanently removes a record and its history. run_report_query takes a free-form sql string and ships no annotations at all. fetch_enrichment lets the caller choose the URL that a customer record gets posted to.

What the checker does, and what it leaves alone

The bundled checker is deterministic and boring on purpose. It handles the MCP handshake over streamable HTTP, calls tools/list, and for each tool records what it claims and what the evidence shows. It never calls a tool on the server under review: listing is safe, invoking is not, and a server that misannotates a destructive tool will also misrepresent the cost of trying it.

Getting the heuristics right mattered more than getting them clever. The first version flagged search_customers as arbitrary execution because its parameter is called query, which is exactly the kind of false positive that teaches people to ignore a tool. A search tool takes a query string. Real SQL gives itself away in the name or the description, so that is where the check looks now.

Run against the candidate, through the gateway:

export ARCTL_API_TOKEN=$(...)
mcp-intake http://records-mcp.localtest.me/mcp --registry http://agentregistry.localtest.me

mcp-intake 1.0.0 · candidate MCP server review
candidate: http://records-mcp.localtest.me/mcp
server:    records-mcp 3.4.6
tools:     6

TOOL                    CLAIMED         EVIDENCE      OPEN WORLD
search_customers        read-only       read          no
get_customer            read-only       read          no
update_customer_email   not read-only   write         no
delete_customer         read-only       destructive   no
run_report_query        unstated        execute       no
fetch_enrichment        unstated        write         yes

FINDINGS
  [HIGH] delete_customer · contradicted-claim
        annotated readOnlyHint=true but the name and description describe a destructive operation
  [HIGH] run_report_query · arbitrary-execution
        takes caller-supplied sql and runs it
  [MEDIUM] run_report_query · unannotated
        ships no annotations, so the server makes no claim to check
  [HIGH] fetch_enrichment · egress
        the caller chooses the destination (url), so this is a data egress path

CATALOGUE OVERLAP
  crm-lookup already exposes: get_customer, search_customers

VERDICT: HOLD (3 high-severity findings)
Tools that pass unconditionally: search_customers, get_customer

The catalogue line is the one that changes the decision. The two tools that survive review are tools the platform already has, served by a server that is already approved. So the question stops being whether this server is safe to admit, and becomes what it adds that is not already available. The answer is run_report_query, fetch_enrichment, update_customer_email and delete_customer, which is the entire set of things the review objected to.

The half a checker cannot do

Pattern matching finds the contradictions. It cannot read meaning, and it cannot reason about tools in combination. That is the division of labour the skill is written around: run the checker, report it verbatim, then add only what it could not determine.

Asked to review the same server inside Claude Code, the model kept the checker's findings and then made three points the checker had no way to reach:

1. A falsified annotation is not a local problem. Once this server has been shown to annotate a permanent delete as read-only, readOnlyHint: true on search_customers and get_customer stops being evidence. The two tools the checker passed are passed partly on a signal the same server has already falsified.
2. The exfiltration path is inside this one server. run_report_query reads the reporting replica and fetch_enrichment posts to a caller-chosen URL. Read then send, one call each, no second backend needed. A per-tool table cannot show a pair.
3. The overlap is duplication, not replacement. The two shared tools have identical descriptions, identical schemas and the same reported version as the approved server.

Its recommendation was to decline rather than gate, on the grounds that gating admits four risky tools in exchange for a second copy of two the platform already has. That is a better answer than the checker's HOLD, and it is the reason to put a model in front of a linter rather than shipping the linter alone.

Publishing it as a plugin

The bundle is deliberately small. A skill carrying the method, a command, a read-only sub-agent that re-reviews catalogued servers for drift, and the checker itself:

There are no hooks here and no .mcp.json. The previous field guide shipped both, and the inventory said so loudly. This one reads and reports, so its inventory is four lines and a reviewer can approve it quickly. That difference is the point of having an inventory at all.

Register the bundle as a pinned pointer, the same as any other plugin:

apiVersion: ar.dev/v1alpha1
kind: Plugin
metadata:
  name: mcp-intake-review
  namespace: default
spec:
  title: MCP intake review
  description: >-
    Reviews a candidate MCP server before it is allowed onto the platform.
    Lists its tools, checks what each one claims about itself against what its
    name, description and schema show, compares against the catalogue, and
    emits the agentgateway policy that would allow only the tools that passed.
  harnesses:
    - claude-code
  source:
    type: git
    git:
      repository:
        url: https://github.com/tjorourke/mcp-intake-review
        branch: main

arctl apply -f yaml/plugin.yaml
claude plugin marketplace update agentregistry
claude plugin install default.mcp-intake-review@agentregistry

/default.mcp-intake-review:mcp-intake http://records-mcp.localtest.me/mcp http://agentregistry.localtest.me

The same review as an agent, with arctl

The laptop half is only one of the two places this needs to run. CI needs the same answer before an MCPServer resource merges, and other agents need to be able to ask for it over A2A. That is an agent, and arctl scaffolds it.

arctl init agent mcpintake \
  --framework adk --language python \
  --model-provider anthropic --model-name claude-sonnet-4-5 \
  --description "Reviews a candidate MCP server and returns a machine-readable intake verdict"

That writes a runnable ADK project: the agent package, a Dockerfile, an agent card, an MCP tool loader, and a build_instruction helper that folds registry skills into the system prompt. Agent names become Python identifiers, so no hyphens and no dots.

The tools are the interesting part. Rather than reimplement the review, the agent imports the same checker the plugin ships and exposes two functions:

def review_mcp_server(url: str) -> dict:
    """Review a candidate MCP server and return the structured intake verdict."""

def generate_gateway_policy(server_name: str, allowed_tools: list[str]) -> str:
    """Return the AgentgatewayPolicy that allows only the named tools."""

One classifier, two surfaces, so a developer and a pipeline cannot reach different verdicts about the same server. The instruction carries the same prohibition as the skill: never call a tool on the server under review.

Point the image at a registry the cluster can pull from, then build and publish:

apiVersion: ar.dev/v1alpha1
kind: Agent
metadata:
  name: mcpintake
  namespace: default
spec:
  title: MCP intake reviewer
  description: >-
    Reviews a candidate MCP server and returns a machine-readable intake
    verdict. Runs the same checks as the mcp-intake-review plugin, so a CI
    gate and a developer laptop reach the same answer.
  source:
    image: localhost:5001/solo-demos/mcpintake:1.0.0
    protocol: A2A

arctl build ./mcpintake --push --platform linux/arm64
arctl apply -f mcpintake/agent.yaml
arctl get agents

→ Injecting labels from arctl.yaml: arctl.dev/framework=adk, arctl.dev/language=python
✓ Agent/mcpintake (latest) created

NAME        TAG      MODE     DESCRIPTION
mcpintake   latest   source   Reviews a candidate MCP server and returns a machine-read...

arctl build reads arctl.yaml to pick the framework's build command and takes the image tag from spec.source.image unless you override it with --image. arctl apply then publishes the catalogue row and stamps the framework and language as labels.

Rolling it out is a Deployment naming the agent and a runtime:

apiVersion: ar.dev/v1alpha1
kind: Deployment
metadata:
  name: mcpintake
  namespace: default
spec:
  targetRef:
    kind: Agent
    name: mcpintake
  runtimeRef:
    kind: Runtime
    name: kagent-local

Everything above through arctl get agents was run end to end. The rollout was not: it needs a registered runtime, and this cluster deliberately carries no kagent install. Treat the Deployment above as the shape rather than as a tested step.

The artefact the review produces

A verdict that ends in prose gets argued with. This one ends in the policy that would enforce it, generated from the tools that survived:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: records-mcp-intake
  namespace: mcp-candidates
spec:
  targetRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: records-mcp
  backend:
    mcp:
      authorization:
        action: Allow
        policy:
          matchExpressions:
            - mcp.tool.name == "search_customers" || mcp.tool.name == "get_customer"

Two details worth knowing. The CEL variable is mcp.tool.name, and it composes with jwt.*, so the same shape narrows further to a per-team or per-agent allow-list rather than a flat one. And backend.mcp.authorization lives on the agentgateway.dev group, so what the review emits applies on OSS as well as Enterprise.

The gateway is doing the enforcing because the gateway is already where these servers are reached. Every MCP endpoint in this lab sits behind agentgateway, which is what makes tool discovery uniform enough to review in the first place: one hostname pattern, one place to attach policy, one audit point.

What to take from this