Open source · MIT · built in the open

Know your LLM feature works before your users do.

Evalstone is the plug-and-play evals layer for any LLM-based project: a Python library, a learning center, a check center, an API and an MCP server. First it teaches you what evals are. Then it runs them for you, in CI and in production.

Start with Evals 101How Evalstone helpsGitHub
evals/support_bot.py
import evalstone as es

suite = es.Suite(
    "support-bot",
    dataset=es.Dataset.from_jsonl("evals/golden.jsonl"),
    checks=[
        es.checks.json_schema(schema=Answer),
        es.checks.judge("helpfulness", rubric=RUBRIC),
        es.checks.rag.faithfulness(),
    ],
    gate=es.Gate(min_pass_rate=0.95),
)

run = suite.run(target=answer)   # your function
suite.enforce(run)               # raises GateFailed → red in CI

See it

One minute, end to end

The CLI catches a regression on a fresh project; then a center grades the judge against people and promotes it. Recorded from real commands and a center running locally.
evalstone init → gate passes → a new prompt regresses, the gate fails (exit 1) → fixed → a judged run in the center → ten labels → Calibrate → Promote.
  • A judged run in the center: per-check pass rates, the failed gate, and the judge reason for every verdict

    A run

    Per-check results, the gate the CLI enforced, and the judge’s reason for every verdict.

  • The labeler: input, expected, output and the machine verdict, with 1–5 grade buttons and keyboard hints

    Labeling

    People grade outputs from the keyboard: 1–5, N for notes, S to skip, Enter to submit.

  • The Judges page: judge settings, the judge versions seen, and a finished calibration with agreement, kappa and a Promote button

    Calibration

    The judge measured against those labels (agreement, Cohen’s kappa), then promoted.

The problem

Shipping LLM features by vibes works until it doesn't

Every team that puts a model in front of users hits the same four walls.

"It looks fine" is not a test

You tweak a prompt, try three questions, and ship. The fourth question, the one a customer asks, is the one that broke. Non-deterministic systems need a repeatable answer to "is it better?".

Model updates move the floor

Providers retire and replace models. The same prompt gives different answers next month. Without a golden set and a baseline run you only find out from support tickets.

A judge you never measured is a guess

LLM-as-judge is powerful and cheap, and it is also biased, position-sensitive and happy to agree with itself. A judge earns trust by being calibrated against humans.

Production is the eval nobody runs

Offline suites catch regressions you anticipated. Real traffic finds the rest. Sampling and scoring live exchanges turns "we think it is fine" into a number with a trend.

Evals, in one minute

An eval is a test for a system that does not give the same answer twice

Unit tests assert equality. Evals measure quality over a dataset and decide whether the number is good enough. The loop is small, and it is the same loop whether you are grading a chatbot, a RAG pipeline, an extraction prompt or an agent.
  1. 1

    Dataset

    A set of inputs with, where possible, what a good answer looks like. Real traffic, hand-written, or synthetic.

  2. 2

    Run

    Send every input through your feature and keep the outputs. Pin the dataset version and the git commit.

  3. 3

    Check

    Score each output: deterministic rules, an LLM judge with a rubric, a human, or a comparison against another run.

  4. 4

    Gate

    Turn scores into a decision: pass rate above a threshold, no regression versus the baseline, cost under budget.

Read the full chapter

How Evalstone helps

One layer, from the first golden set to production monitoring

Everything is designed to be plugged into an existing project with no rewrite: a pip install, a YAML file, and a base of checks you extend in plain Python.

Learning center

live
A short course on evals: what they are, why they matter, the kinds of checks, judges, gates, production, and the pitfalls. Read it before you write a single test.

Library

live
Datasets, checks, suites, runs and gates as plain Python. Deterministic checks, LLM judges, RAG, safety, pairwise and agent-trajectory checks built in.

Training center

live
Versioned golden sets, a human labeling queue, and judge calibration so your LLM judge agrees with your people.

Check center

live
Run history with baselines and diffs, a catalog of reusable checks, and alerts when a monitor sees quality drift.

CI gate

live
A GitHub Action and a pytest plugin. A pull request that lowers the pass rate or regresses the baseline goes red.

Production monitoring

live
Sample live exchanges, score them continuously, and see the trend. Works with any proxy or SDK that can post a trace.

API

live
Everything the center does is a REST endpoint with OpenAPI docs, so other languages and other tools can push runs and read results.

MCP server

live
Let Claude Code, Cursor or any MCP client run a suite, judge an output or record a sample while you work.

How it works

Three steps, no rewrite

  1. 1

    Describe the suite

    One YAML file: where the dataset is, what produces outputs (a function, an HTTP endpoint, a prompt), which checks to run, what the gate is.

  2. 2

    Run it anywhere

    evalstone run on your laptop, in pytest, in a GitHub Action, or from an MCP client. The judge scores quality with the model you name, on your key or the center's; deterministic and trajectory checks run before it in every suite.

  3. 3

    Keep the history

    Point it at a center, self-hosted or ours, and every run lands in PostgreSQL: baselines, diffs, labels, calibrations, monitors.

evalstone.yaml
# evalstone.yaml
project: support-bot
datasets:
  golden: evals/golden.jsonl
suites:
  nightly:
    dataset: golden
    target: { kind: http, url: http://localhost:8000/answer }
    checks: [json_schema, helpfulness, faithfulness]
    gate: { min_pass_rate: 0.95, max_regression: 0.02 }
judges:
  default_model: anthropic/claude-haiku-5-5
center:                      # optional — persists runs in PostgreSQL
  url: https://api.evalstone.com
  project: <project id>
  api_key_env: EVALSTONE_API_KEY

Where the model comes in

Quality is judged by a model you choose. Contracts are checked exactly. Humans are the ground truth.

Three layers, and the judge is where quality gets measured: a rubric, a small scale, a model you name, checked against labels people wrote. Around it, deterministic checks catch the contract failures exactly and for free. Standalone, the judge calls your provider directly, with your key; attached to a center, it goes through the center's proxy on the organization's key.

1 · the quality signal

LLM-as-judge, with your model

  • A rubric on a small scale, graded by the model you name; the verdict is a level and a reason, as JSON
  • rag.faithfulness, rag.context_relevance, rag.answer_relevance: judges with fixed rubrics
  • comparison.pairwise: which of two outputs is better, both orders, so position bias cancels
  • Measured against human labels: a judge you have not checked is an opinion (the agreement report — kappa, confusion — is in Evalstone Center)
  • No key, a refusal, malformed JSON: an error, never a pass, and the gate fails on error rate

2 · the ground truth

Human labels

  • evalstone label grades outputs in the terminal, resumably, keyed by sample and output
  • The human check scores from those labels; they are the ground truth judges are calibrated against

3 · the guardrails

Deterministic checks, no model needed

  • JSON schema, must-include facts, regex, length, numeric tolerance
  • Safety rules (refusal, PII, forbidden phrases) and agent-trajectory checks
  • Exact and free; deterministic and trajectory checks run before the judges in every suite
evalstone.yaml
checks:
  helpful:
    type: judge                                 # the quality signal
    rubric: |
      Answers the question from the policy; states the concrete rule;
      invents nothing that is not in the policy.
    scale: 5                                    # pass at 4
judges:
  default_model: anthropic/claude-haiku-5-5     # provider/model; switch here, or per check
  # default_model: openai/anthropic/claude-haiku-5-5   # the same model through OpenRouter
providers:                                      # keys come from the environment
  anthropic: { api_key_env: ANTHROPIC_API_KEY }
  openai:    { base_url: https://openrouter.ai/api/v1, api_key_env: OPENROUTER_API_KEY }
  gemini:    { api_key_env: GEMINI_API_KEY }

Three native adapters: anthropic through the official SDK, gemini through Google's API, and openai, which speaks the Chat Completions API to api.openai.com or to any compatible server you point base_url at, so OpenRouter gives you every model it hosts and Ollama runs one locally. Switching the judge is one line (default_model, or model: on a single check). Keys are read from environment variables and never written to a report. Verdicts are cached by content for the life of the process; calls to models in the price table are priced; a missing key or a failed call is an error, never a pass.

How Evalstone works, in fullWhy judges need calibrating (Evals 101, ch. 6)

Learning center

Evals 101, nine short chapters

Written for people who ship LLM features and have never run an eval. Each chapter explains why, shows the pitfall, then the practice. Read in order or jump in.

1 · What are evals?

Tests for non-deterministic systems, and why "it looks fine" is not one.

2 · Why you need them

The regressions you did not see: model updates, prompt drift, and silent cost.

3 · The eval loop

Dataset → run → inspect → fix → re-run → gate. Small loop, run often.

4 · Kinds of checks

Deterministic, LLM-as-judge, human, pairwise, RAG, agent trajectory, safety.

5 · Building a golden set

Harvest, label, split, version. How big is big enough.

6 · LLM-as-judge done right

Rubrics, scales, position bias, calibration against humans, cost control.

7 · Gates and CI

Thresholds, baselines, flakiness, and what to actually block on.

8 · Evals in production

Sampling, monitors, drift, and avoiding alert fatigue.

9 · Pitfalls

Overfitting the golden set, leaking answers, judge collusion, green-but-broken.

Status

Built in the open, in this order

  1. live

    Site + learning center

  2. live

    Library + CLI on PyPI

  3. live

    MCP server · pytest · labeling · GitHub Action

  4. live

    Center: API + web, self-hosted or hosted

The roadmap, the design decisions and every commit are public. Star the repository to follow along, or open an issue with the eval problem you are stuck on; that is how the chapters get better.

Follow on GitHubRead the docs