Changelog

Every release of the evalstone library (PyPI), newest first; semantic versioning, tags lib-vX.Y.Z in the repository. Install or upgrade with pip install -U evalstone.

0.2.4 — 2026-10-14Copy Anchor Link

A gate can require its baseline (suggested in feedback on the launch post).

• Gate(require_baseline=True) (YAML: gate: { max_regression: 0.02, require_baseline: true }) fails the gate when the regression comparison cannot run: no baseline, a baseline from another suite, one that scored nothing or failed its own max_error_rate (recorded during a judge outage), or one from a different dataset version or check set. Without it nothing changes: a missing baseline is SKIP regression: no baseline run, a different dataset version is a WARN, and the gate can still pass. Use it in CI once a baseline report exists, so an expired artifact or a wrong --baseline path fails the job instead of turning the regression check off.
• require_baseline without max_regression is a configuration error.
• A change that edits the dataset or the check list has no comparable baseline, so under require_baseline it fails until the baseline is refreshed (rerun the suite on the new definition and save that report as the baseline). Checks are compared by key: a check that keeps its key but changes its threshold or judge is not detected.
• Gates reject unknown keys: a misspelt threshold (min_pass_rte: 0.9) is now a configuration error (exit 2) instead of being ignored.

0.2.3 — 2026-10-14Copy Anchor Link

Promote pins the model the verdicts were routed through, found on the demo project's first real calibration.

• judge and comparison.pairwise record raw.judge_model: the <provider>/<model> the verdict was routed through (for example openrouter/anthropic/claude-haiku-5.5), next to raw.model, the id the provider reported (anthropic/claude-haiku-5.5). Verdicts served from the cache carry it too.
• The center stamps only raw.judge_model, so the Judges page, calibrations and Promote use the routed model. Before, it stamped raw.model, and Promote sent that provider-reported id, which judge settings reject (no provider named 'anthropic') or, when a provider of that name existed, route elsewhere. Results pushed by an older library are no longer stamped (a migration clears the old stamps): a version seen only from them shows no model and cannot be promoted until a 0.2.3+ run pushes it.

0.2.2 — 2026-10-14Copy Anchor Link

Judge replies cut off by the token cap, found by the first run against a real vendor (Claude Haiku 5.5 via OpenRouter on the rag-bot demo: 4 of 10 verdicts errored).

• judge and comparison.pairwise default max_tokens is now 1024 (was 256). The cut replies needed 286–349 tokens: a verdict plus a long one-sentence reason. max_tokens is not part of the judge version, so cached verdicts stay valid.
• A reply the provider stopped at the cap (length, max_tokens, MAX_TOKENS) whose JSON is unclosed or invalid now errors with "reply truncated at max_tokens=N; raise the check's max_tokens" followed by the original parse error and excerpt instead of only "no JSON object in judge reply". A complete verdict at the cap is accepted, and a complete but wrong one (score out of range) keeps its own error. ChatResponse.truncated exposes the check.
• comparison.pairwise: an unparseable judge reply now counts that call's cost (and latency) in the errored score, as judge already did.

0.2.1 — 2026-10-14Copy Anchor Link

Attach fixes found by the docs audit.

• comparison.pairwise built its check version (and so its cache key) from --judge-model or the check's own model:, ignoring the attached project's per-check override — the call itself already honoured it. Both now resolve through the same precedence (--judge-model > center override > model: > default), so a verdict cached under the override is found again and a changed override invalidates the cache.
• Docstrings and this changelog no longer claim prompt targets are proxied when attached; they never were (providers: + env keys serve the system under test).
• Known limitation, documented: when attached, --judge-model some-typo/model is not validated up front — the center only rejects the provider at the first judge call, which surfaces as an errored score. Standalone runs still fail fast.

0.2.0 — 2026-10-14Copy Anchor Link

Attach a project to a center.

• center: in evalstone.yaml (url, project, api_key_env — an es_… ingest key) routes judge calls through the center's model proxy, so no judge key lives on the machine running the suite; the project's judge settings pick the model (per-check overrides, then the default), --judge-model still wins.
• Judge verdicts are cached in the center (CenterCache); a center hiccup is a cache miss, never a failed run.
• human checks of an attached project look labels up in the center (labels: center, the default when attached; a file path still works).
• evalstone run pushes its report to the center (push_runs: false to opt out); a failed push warns and leaves the exit code alone.
• Prompt targets (the system under test) always use the local providers: + env keys, even when attached — the center proxy is for judges only. The MCP server honours the attach too.

0.1.1 — 2026-10-13Copy Anchor Link

Judge switching and cost honesty.

• Named providers: a providers: entry may name any adapter (adapter: openai | anthropic | gemini), so openrouter:, ollama: and openai: can coexist and a judge model reads openrouter/anthropic/claude-haiku-5-5. Unknown provider names and adapters are reported with the known ones. A named entry never falls back to the adapter's default key variable (one vendor's key is never sent to another); a 401/403 without a key says so.
• --judge-model provider/model on evalstone run and gate overrides every judge's and pairwise comparison's model for that run — never the target under test (Context.judge_model; Config.context(judge_model=…)). Validated before the run starts; a typo exits 2.
• short_circuit: true on a suite skips the judge checks on a sample that a contract check (deterministic, safety, trajectory — not a human label, which is a quality verdict the judge should still be compared against) already failed. The skipped result is a failure, not an error (Score.skipped, RunSummary.skipped, CheckSummary.skipped), so no paid call is spent on an output that already failed and the error-rate gate is not tripped. Off by default. Turning it on changes pass rates (a skipped judge counts as a failure), so re-take your baseline when you enable it.
• Cheap-first ordering now includes safety rules and human labels, not only deterministic and trajectory checks.
• Gate: with max_cost_usd set, a WARN line counts results from models outside the price table (they count as $0). Gemini models priced from the official table (2026-10-09); OpenAI models are set under prices:.

0.1.0 — 2026-10-13Copy Anchor Link

The first usable release: the hero path from evalstone.yaml to a gated run.

• Core: Sample / Dataset (JSONL, content-hashed versions, deterministic filter / sample / split), Output, Score (an error is never a pass), the Check protocol, @check, the registry.
• Deterministic checks: exact, contains, regex, json_schema (JSON Schema or pydantic), numeric_tolerance, length, python.
• Targets: callable, http, recorded, prompt.
• Suite / Run / Gate: per-check and per-tag summaries, dataset hash + git SHA provenance, baseline regression gate, error-rate gate (default 0), JSON report, compare.
• CLI: init, run, gate, compare, show, list.
• Providers: OpenAI-compatible (OpenAI, OpenRouter, Ollama, vLLM), Anthropic (official SDK, evalstone[anthropic]), Gemini; pricing table with overrides; unknown models reported as unpriced.
• LLM-as-judge: rubric → small scale → JSON verdict, reference excluded unless asked, judge version hash, content-keyed cache.
• RAG (faithfulness, context relevance, answer relevance), safety (refusal, no_pii, forbidden), pairwise comparison (both orders), trajectory (tool_called, tool_order, max_steps, no_repeated_calls, finished).
• Sample: samples/rag-bot — a no-LLM retrieval bot with a gated suite.
• MCP server (evalstone mcp, extra evalstone[mcp]): list_suites, run_suite, judge_output, record_sample, compare_runs, get_run over stdio; calls run one at a time, stray stdout goes to stderr, one provider registry and judge cache per session, report paths relative to the config.
• pytest plugin (entry point pytest11): @suite("name"), fixtures evalstone_run / evalstone_gate / evalstone_config, options --evalstone-config and --evalstone-baseline (a report or a directory of them); one report per suite (evalstone-report.<suite>.json); a suite without a gate fails the gated forms; the package imports lazily so the plugin costs nothing in unrelated test sessions.
• GitHub Action (action/): run a suite on a pull request, job summary, in-place PR comment (evalstone show --markdown), fail on the gate; inputs bound through the environment; report output is an absolute path.
• Human labels: evalstone label <report> grades outputs in the terminal (pass/fail or a 1..N scale, resumable, keyed by sample id + output hash); the human check scores from the labels file and errors on unlabelled outputs.
• Reports: Run.markdown(), Run.failure_lines(), results carry the sample's input and expected value.
• Loading: project modules are reused across runs in one process unless a file changed or the registry was cleared, so test patches reach suites.