How Evalstone helps
The pieces of Evalstone and how they map onto the practices in Evals 101.
Status
The site and the learning center are live. The library, the integrations and the center are being built in the open, in that order. Pages in this section describe the design; each one says at the top whether its piece exists yet.
Evals 101 describes a practice: a golden set, a loop, checks you can trust, a gate, and production monitoring. Evalstone is that practice as software you can add to an existing project without rewriting it.
The pieces
| Piece | What it does | Maps to |
|---|---|---|
Library (pip install evalstone) | Datasets, checks, suites, runs, gates as plain Python. A CLI. | chapters 1, 3, 4, 7 |
| Checks catalog | Built-in deterministic, judge, RAG, safety, pairwise and trajectory checks, plus yours. | chapter 4 |
| Training center | Versioned golden sets, harvesting from traffic, a labeling queue, judge calibration. | chapters 5, 6 |
| Check center | Run history, baselines and diffs, catalog, monitors and alerts. | chapters 7, 8 |
| CI gate | GitHub Action + pytest plugin. | chapter 7 |
| Production monitoring | Trace ingest, sampled scoring, trend, alerts. | chapter 8 |
| API | Everything the center does, as REST with OpenAPI docs. | — |
| MCP server | Run suites and judge outputs from Claude Code, Cursor, any MCP client. | — |
The one design rule
The library computes; the center stores. The library has no database. A run lives in memory, prints a table and writes a JSON report. When you point it at a center (yours, self-hosted, or the hosted one) every run, result, label and judge-cache entry is persisted in PostgreSQL through the center's API, which is the only process that touches the database. One storage engine, one place to reason about data.
Plug and play
- No SDK rewrite. The target of a suite is your function, your HTTP endpoint, or a prompt + model. Evalstone calls it; it does not wrap it.
- One YAML file describes datasets, suites, checks and gates. The same file runs locally, in pytest, in CI and from MCP.
- Checks are plain functions. The built-ins have the same shape as the ones you write.
- Honest numbers. A judge error is never a pass. Every run reports cost. Runs pin a dataset version and a git SHA so any two runs compare.
Where to go next
Concepts
Dataset, sample, check, suite, gate, target, run, result, label, judge version, monitor.
Library
The Python API and the YAML file (design preview).
Center
Training center + check center: API and web (design).
MCP server
Evals from inside your coding agent (design).
Roadmap
What is built, what is next.