CATT

Consumer Agent Testing Tool (CATT)

Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.

Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.

Contributing | Security | Changelog | MIT license

A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented, unofficial, assumed or unknown. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.

What you can test today

The connector lane records the basis for each Muse-specific behavior. Tool classes and approval prompts follow documented guidelines. The muse-cli projection and template client follow unofficial mirrors; the native projection is an assumed generic baseline. The neutral cli projection supports both targets as a CATT treatment. Instinct runs have a separate source record and label the shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter; no public Instinct MCP integration contract was found.

Run the local booking example under Instinct without a model or account:

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json \
  --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"

Set emulation_target: instinct in your scenario or pass --emulation-target instinct. Both targets use native or cli projection and the generic MCP client. Instinct defaults to plain-text replies and does not run Muse guideline checks. Its local wallet fixture covers payment behavior; live payment rails, trusted-network protocol, Vault and Files remain outside the shared contract.

Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.

Setup

You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), uv, and Node.js 20 or newer with npm (the installed packages declare >=20; tested with 25.9). The recorded release evidence covers macOS. From a clone of this repository, install the Python and Node dependencies and the Chromium build the browser lanes use:

uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium

The connector lane, the readiness checks and the scripted runs need only uv sync. Node and Chromium are for the MCP Apps and browser lanes.

On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.

sudo apt-get update && sudo apt-get install -y nodejs npm   # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
  node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium

Three features rely on macOS Seatbelt (sandbox-exec) and report unsupported or blocked on Linux rather than running unconfined: the --offline readiness mode, the network-denied validator workers behind run --surface api-tools, and the local-model memory profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it.

Check the setup with doctor:

uv run consumer-agent-testing-tool doctor

It checks Node against the floors the installed packages declare, the tooling/payment-ui packages against their pinned versions, and Chromium and its headless shell under .cache/ms-playwright. It reports the optional protocol tooling, and checks Docker only for development lanes or configured stdio MCP and CLI targets. It makes no model requests, even when an existing configuration or --model-profile selects a model. Each failed or blocked check prints one line on stderr with its reason and, for a setup problem, the command that fixes it.

To explicitly run the provider preflight, which sends two short model requests and may incur charges, add --model-preflight:

uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight

Scripted runs need no model account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a model profile and an API key; Model providers shows how to create one. Supply credentials through environment variables, and keep them out of scenarios and shared evidence. consumer-agent-testing-tool.yaml, the default settings file, is local and git-ignored, and no profile is shipped.

Quickstart: connector lane

This quickstart stays on loopback and needs no account. Complete Setup, then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none

Replay the example's scripted calls, with no model:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json

The script searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's private: values, which the model never sees, and the scripted user answers each approval card from the scenario's permissions: rules. After the run, the host checks the fixture's private oracle (one confirmed booking, no duplicate charges) and the booking's status through get_booking. It also checks that each sent call matched its approval card (approval.dispatches_match, approval.disclosure_bound) and that the final answer made no false claim and relayed the total and the confirmation (answer.no_false_claims, answer.relay_complete).

The command prints the run directory and Connector report: .../report.html, and exits 0 when every required check passed. Runs are sealed under runs/connector-<id>/. Share the output of export, not the run directory: its private/ folder holds host-private values.

For an agent run, create agent.yaml with the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

The model preflight runs first and prints its own evidence path. The agent then sees the four allowed tools as native functions, with the host-private email removed from their schemas, and writes the other arguments itself. This uses paid model inference. To change the treatment without editing the scenario, --projection cli offers the same tools through one exec command for either target, --exposure search-only loads their parameters through tool_search, --output-schema strict refuses results that break a tool's outputSchema as the official MCP SDK does, and --trials N runs independent trials and prints pass^k. The run lock records the projection, exposure, client-profile and output-schema overrides under overrides.

connector-run exits 0 when every required check passed; 1 when a check failed, the answer made a false claim or an agent limit was reached; 2 for a setup, harness or reporting error; and 3 when the trial is blocked or unsupported, for example by an uncertain write to reconcile, an unobservable check or a client-profile limit. To start from your own server, run scaffold; to replay a recorded, shareable agent trial without a model, turn it into a calls file with scaffold --from-run. The connector lane reference covers scenarios, flags and evidence. The examples list each shipped scenario with its fixture command, and lane extensions covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions.

Quickstart: a popular MCP server

The connector lane works with any MCP server over Streamable HTTP, not only the bundled test service. This quickstart points it at "Everything" (@modelcontextprotocol/server-everything, MIT), the reference server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no account or key. Start it in a second terminal; the first run downloads the pinned release from npm:

PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp

See what the host infers from the live catalog. scaffold runs the handshake and tools/list, calls no tool, and writes a scenario skeleton with suggested tool classes from the server's annotations:

uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines

The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --projection muse-cli --client-profile muse-template-unofficial-2026-10 \
  --driver scripted --script examples/connector/everything-weather-cli.calls.json

Both exit 0. The scenario offers the model three of the server's tools; a reviewed allowlist keeps the rest off its list, including get-env, which returns the server process's environment variables, and gzip-file-as-resource, which fetches any URL it is given. The server has no private oracle, so the outcome check calls get-structured-content itself after the agent stops, and the answer checks look for the temperature and conditions the server returned. With a model profile, let an agent do the same task three times and print pass^k (paid inference). The scenario declares no reset URL, so connector-run warns that the trials share the server's state; this weather tool keeps none:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --config agent.yaml --trials 3

Readiness checks

check also runs readiness suites that read catalogs and documents instead of running an agent. With the connector-lane quickstart's fixture still running, review its catalog and the example manifest against the pinned Muse connector guidelines:

uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \
  --manifest examples/connector/readiness.yaml

The guidelines review lists the server's tools and calls none; --drift-check fetches the live guidelines, and only when given. Poppy discovery only reads. Its probes run only with --manifest and act on the company's own hosts: they start sessions, sign in the declared test account and confirm one declared operation. The gadget suite reads one local file. The contract suite checks a target named in the configuration file (--config). For a kind: mcp target without declared contract_cases, its operation-contracts check is unsupported and the run exits 3.

A clean readiness result is not an admission. Meta reviews connectors by hand, Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms say it is not a developer platform. Readiness checks describes the rules, manifests, --offline behavior and the mcp-contract-2026-07-28 profile.

Quickstart: Coverage Cat sandbox

From a source checkout or unpacked source distribution, complete Setup, then run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted

The example creates a fictional shopper's draft in Coverage Cat's hosted sandbox at https://www.coveragecat.com/api/consumer/mcp. It needs no Coverage Cat account, operator credentials or model API key.

The first umbrella_consumer_prefill call sets sandbox: true, as