Evals from your coding agent
The MCP server lets Claude Code, Cursor or any MCP client run your suite before it says it is done.
If an agent is editing your prompt, it should run your evals. The Evalstone
MCP server exposes a project's evalstone.yaml as tools over stdio.
Setup
pip install "evalstone[mcp]"
cd your-project # where evalstone.yaml lives
claude mcp add evalstone -- evalstone mcpOther clients: run evalstone mcp as a stdio server from the project directory.
Tools
| Tool | What it does |
|---|---|
list_suites | project, datasets, checks (configured and @check-registered), suites with their gates |
run_suite(suite, report?, baseline?, failures?) | runs the suite; returns summary, per-check rates, gate verdict and the failing results with reasons; writes the JSON report |
judge_output(check, output, input?, expected?) | scores one answer with any configured check — a judge, a schema check, a safety rule |
record_sample(dataset, input, expected?, tags?, id?) | appends a case to a dataset JSONL (source: harvested); duplicates are refused |
compare_runs(a, b) | regressions, improvements and metric deltas between two reports |
get_run(report, failures?) | reads a saved report |
Errors come back as {"error": …} results the agent can read, never as protocol
failures. The YAML is reloaded on every call, so editing a check or a dataset
takes effect without restarting the server; the provider clients and the judge
cache live for the whole session. Report paths are relative to the directory
holding evalstone.yaml, wherever the server was started from (evalstone mcp --config path/to/evalstone.yaml serves a project from elsewhere). Calls run one
at a time, and anything a check prints goes to stderr, never onto the wire.
Attached to a center
When the YAML has a center: block the server behaves like the CLI: judges go
through the center's model proxy with the project's judge settings, verdicts
come from and go to the center's shared cache, human checks look labels up
there, and EVALSTONE_API_KEY must be in the environment the client starts the
server with. Prompt targets (the thing under test) still use the local
providers: registry. run_suite never pushes a run to the center: the
report it writes is the record, and runs it makes carry trigger: "mcp".
A typical loop
list_suites— what exists.run_suite("nightly")— 87.5%, gate failed, five failures listed.- The agent edits the prompt or the retriever.
run_suite("nightly", report="b.json")→compare_runs("evalstone-report.json", "b.json").- It reports the diff to you. The gate stays yours.