Evals in CI

Pick the lightest one that fits. All three run the same suite and the same gate.

The exit codeCopy Anchor Link

evalstone gate nightly --baseline reports/main.json

Exit 1 when the gate fails. --baseline supplies the run that a max_regression: threshold in the suite's gate compares against (a saved report from main, say); a gate without max_regression ignores the baseline and prints no regression line. Without a baseline the regression check is skipped and the gate can still pass, which suits a first run; once main has a baseline report, add require_baseline: true to the gate so a missing or mismatched baseline fails the job instead. A PR that edits the dataset or the check list then fails too, since nothing comparable exists yet: refresh the baseline (rerun the suite on the new definition and save that report) as part of that change. Works in any CI.

The pytest pluginCopy Anchor Link

Installed with the library (no extra), registered automatically.

# tests/test_evals.py
from evalstone.pytest_plugin import suite

@suite("nightly")                 # runs the suite; a failing gate fails the test
def test_support_bot(run):        # optional: the body sees the Run
    assert run.summary.cost_usd < 1.0

The failure message carries the gate lines and the first failing results, so the test output already answers "which sample broke". Fixtures evalstone_run and evalstone_gate do the same without the decorator, and the decorator works on test methods too. Each suite writes its own report beside the config (evalstone-report.<suite>.json), so several suites in one session never overwrite each other; --evalstone-config and --evalstone-baseline (a report, or a directory of per-suite reports, relative to the config's directory) are pytest options. A suite without a gate: fails the gated forms rather than passing by default. Patches you make in a test (monkeypatch on your bot module) reach the suite: the run reuses the modules already imported. No evalstone.yaml → the test is skipped, not errored.

When the project is attached to a center, the plugin judges through the center's proxy and cache like the CLI does, but it never pushes the run: the per-suite report on disk is its output. Runs it makes carry trigger: "ci"; evalstone run (and so the GitHub Action) records trigger: "cli", the MCP server "mcp", so a center can tell them apart.

The GitHub ActionCopy Anchor Link

# .github/workflows/evals.yml (the minimum; every input is in the action's README)
permissions: { contents: read, pull-requests: write }
steps:
  - uses: actions/checkout@v4
  - uses: marceloribeiro/evalstone/action@action-v0   # the floating major tag; action-v0.1.0 pins a release
    with:
      suite: nightly
      working-directory: evals          # where evalstone.yaml lives
      install: "evalstone[anthropic]"   # a pip spec: extras, a pin, a git URL or a path
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # standalone judges
      # EVALSTONE_API_KEY: ${{ secrets.EVALSTONE_API_KEY }} # attached: the ingest key instead

The action installs evalstone (install: is a pip spec: the published name with extras or a pin, a git URL, or a path inside a monorepo), runs the suite, writes the report into the job summary, posts a PR comment (updated in place on every push) with the verdict, the per-check table and the failing samples, and fails the check when the gate does not pass. The comment needs a token that can write to the pull request: on pull requests from forks the token is read-only, so the comment is skipped and the job summary is the report. The comment is evalstone show <report> --markdown, so you can see it locally.

The step runs evalstone run, so an attached project (a center: block in its YAML) pushes every Action run to the center; put the project's ingest key in env: as EVALSTONE_API_KEY and drop the provider key — the center's judge settings pick the model. Without the key the step exits 2 before any sample runs — attached, the center client is what judges. A push that fails after the run (the center unreachable, say) only warns and leaves the exit code alone. There is no judge-model input: the model is the YAML's, or the center's.

Outputs: report (the absolute report path, usable as a later run's baseline) and passed (true, false, or none when the suite has no gate). The same table with examples: action/README.md.

The Evalstone repository runs this action on its own pull requests against the rag-bot sample.

What to block onCopy Anchor Link

Chapter 7 in one line: block on deterministic and calibrated checks with a tolerance above your noise floor; report the rest.