Quickstart

Install

pip install "evalstone[anthropic]" — Python 3.12+. The [anthropic] extra is what step 5's judge needs when you run standalone; everything before it runs with no model, and an attached project (step 6) needs no extra at all.

1. Scaffold a project

evalstone init my-evals
cd my-evals

This writes five files. If any of them already exists the command refuses and writes nothing (exit 2), so an existing evals/ package is never touched.

File	What
`evalstone.yaml`	datasets, checks, suites, gate
`evals/golden.jsonl`	two sample questions with references
`evals/outputs.jsonl`	recorded answers, so the first run needs no model
`evals/checks.py`	a custom check (`no_apology`) registered with `@check`
`evals/__init__.py`	makes `evals` importable (`imports: [evals.checks]` in the YAML)

2. Run the suite

evalstone run nightly

You get a per-check table, the failing results with reasons, the gate verdict, and evalstone-report.json. Break a recorded answer in evals/outputs.jsonl and run again: the gate fails and the report names the sample.

3. Point it at your own feature

Replace the recorded target with the thing you ship:

suites:
  nightly:
    dataset: golden
    target: { kind: callable, fn: "myapp.bot:answer" }   # or http / prompt
    checks: [must_include, short, no_apology]
    gate: { min_pass_rate: 0.9 }

callable calls your Python function with the sample input; http POSTs it to an endpoint; prompt runs a template against a model. See the config reference.

4. Gate it

evalstone gate nightly      # exit 1 when the gate fails

Put that in CI, or use the pytest plugin or the GitHub Action.

5. Add a judge

Deterministic checks are the floor. For quality you add an LLM judge with a rubric (read chapter 6 first):

checks:
  helpful:
    type: judge
    rubric: |
      The answer must directly address the question using the policy and state
      the concrete rule (numbers, time windows, conditions).
    scale: 5
judges:
  default_model: anthropic/claude-haiku-5-5     # standalone: needs ANTHROPIC_API_KEY

Standalone, without a key or without the [anthropic] extra installed, a judge is an error, never a pass, and the gate fails on error rate. That is deliberate.

6. Attach a center

The same suite, with its runs kept and its judges paid for by the organization's key instead of one on your laptop: add a center: block and an ingest key.

center:
  url: https://api.evalstone.com        # or your own
  project: <project id>                 # from the project's Overview page

EVALSTONE_API_KEY=es_…  evalstone run nightly
# … center → run <id> stored (https://api.evalstone.com)

Attached, judges: is ignored and providers: no longer serves the judges (a prompt target — the system under test — still uses it): the judge model comes from the project's judge settings in the center, the verdict cache is shared across machines, and every evalstone run pushes its report. The center page explains what moves where; the seven center: keys are in the config reference; the web app is toured in the center, page by page.

1. Scaffold a projectCopy Anchor Link

2. Run the suiteCopy Anchor Link

3. Point it at your own featureCopy Anchor Link

4. Gate itCopy Anchor Link

5. Add a judgeCopy Anchor Link