evalstone.yaml

project: support-bot
imports: [evals.checks]              # modules whose @check registrations to load
report: evalstone-report.json        # default report path (relative to this file)

datasets:
  golden: evals/golden.jsonl         # JSONL: {"id"?, "input", "expected"?, "tags"?, "metadata"?}

checks:                              # keyed definitions: {type: <check type>, ...kwargs}
  must_include: { type: contains }                          # expected.must_include
  schema: { type: json_schema, schema: { type: object } }
  short: { type: length, max_words: 120 }
  helpful: { type: judge, rubric: "...", scale: 5 }
  faithful: { type: rag.faithfulness }
  reviewed: { type: human, labels: evals/labels.jsonl }   # or labels: center (the default when attached)

suites:
  nightly:
    dataset: golden
    target: { kind: callable, fn: "myapp.bot:answer" }
    checks: [must_include, short, helpful, no_apology]      # keys above or @check names
    gate: { min_pass_rate: 0.95, max_regression: 0.02, max_cost_usd: 2, max_error_rate: 0 }
    short_circuit: true              # skip the judges on a sample a contract check already failed

judges:
  default_model: anthropic/claude-haiku-5-5

providers:
  anthropic:  { api_key_env: ANTHROPIC_API_KEY }
  openai:     { api_key_env: OPENAI_API_KEY }
  openrouter: { adapter: openai, base_url: https://openrouter.ai/api/v1, api_key_env: OPENROUTER_API_KEY }
  ollama:     { adapter: openai, base_url: http://localhost:11434/v1 }
  gemini:     { api_key_env: GEMINI_API_KEY }

prices:                              # USD per 1M tokens (input, output); overrides the built-in table
  my-model: [1.0, 2.0]

center:                              # optional: attach the project to an Evalstone center
  url: https://api.evalstone.com
  project: 035d2060-80b1-4a90-9685-29a793df0f65
  api_key_env: EVALSTONE_API_KEY     # an es_… ingest key from the project's settings

providers.<name> takes adapter (openai, anthropic or gemini; defaults to the entry's name), base_url, api_key_env, api_key, headers, timeout (and max_tokens_param for OpenAI-compatible servers). A judge model is <name>/<model>, so openrouter/anthropic/claude-haiku-5-5 and ollama/llama3.1 can sit next to openai/… and anthropic/…; evalstone run --judge-model <name>/<model> overrides every judge (never the target under test) for one run. A named entry needs its own api_key_env or api_key: only an entry named after its adapter (openai:, anthropic:, gemini:) falls back to that adapter's default variable, so one vendor's key is never sent to another (Ollama needs none). The openai adapter speaks to any OpenAI-compatible server, so OpenRouter, Ollama, vLLM or Groq are a base_url away. Keys come from the environment variable named by api_key_env (or the provider's default one); api_key sets one inline, which is for programmatic use — a key in a committed YAML is a committed key. Only models in the built-in price table (Claude models today) or under prices: count toward max_cost_usd; others are reported unpriced, and a gate with max_cost_usd prints a WARN … unpriced model counted as $0 line next to its verdict rather than guessing a price. The full picture: How it works.

Center

center: attaches the project to an Evalstone center (ours at https://api.evalstone.com, or a self-hosted one). Attached, judges run through the center's model proxy with the organization's keys, verdicts are cached there, human checks look labels up there and evalstone run pushes its report. Standalone (no block) nothing leaves the machine.

Key	Default	Meaning
`url`	required	the center's API base URL
`project`	required	the project id (uuid): the `p/<id>` segment of the project's URL in the web app
`api_key_env`	`EVALSTONE_API_KEY`	the environment variable holding an `es_…` ingest key (minted on the project's Settings page by an admin; a user token works too); a missing value is exit 2
`timeout`	`120`	seconds a judge call may wait on the provider behind the proxy
`cache_timeout`	`10`	seconds for cache and label lookups; a slow center is a cache miss, never a failed run
`push_runs`	`true`	`evalstone run` / `gate` push the report; a failed push warns, never fails
`cache`	`true`	read and write judge verdicts in the center; `false` keeps an in-process cache

What changes when attached. judges: is ignored and providers: no longer serves the judges: the judge model comes from the project's judge settings in the center, and no judge key is needed on the machine running the suite. The system under test does not: a prompt target still uses the local providers: entry and its environment key, and callable/http targets are untouched. human checks default to labels: center (write labels: center explicitly to say so; unattached the default is labels.jsonl next to the config, and a file path still works attached). The pytest plugin and the MCP server use the center's judges and cache but never push runs.

Which model a judge uses, first match wins:

evalstone run --judge-model <provider>/<model> (the whole run)
the center project's per-check override (Judges page, keyed by the check's key)
the check's own model:
the center project's default judge model when attached, else judges.default_model

Standalone, step 2 does not exist and an unknown provider in --judge-model is exit 2; attached, the proxy only rejects a bad name per call (see CLI).

Targets

`kind`	Keys	Produces
`callable`	`fn: "module:function"` (imported from the config's directory), `whole_sample: false`	`fn(sample.input)` or `fn(sample)`; may return a value or an `Output`
`http`	`url`, `method: POST`, `headers`, `body` (template; `"{input}"` leaf = raw input, embedded = text), `response_path` (dotted, digits index lists), `timeout`	the JSON response (or the path into it)
`recorded`	`path: outputs.jsonl` (`{"id", "output"}` per line) or `outputs: {id: output}`	offline outputs; missing id = error
`prompt`	`model` (required — never the judge's by default), `template` (`{input}` and input keys), `system`, `max_tokens`	the model's text; cost and tokens recorded

Gate

Key	Meaning
`min_pass_rate`	fraction of results that must pass
`min_mean_score`	mean 0..1 score across results
`max_regression`	allowed drop in pass rate vs `--baseline` (warns when the dataset version differs)
`require_baseline`	default false. With `max_regression`: fail when there is no baseline, or it is from another suite, scored nothing, exceeded `max_error_rate`, or ran on a different dataset version or check set (default: skip or warn)
`max_cost_usd`	total run cost
`max_error_rate`	default 0: a judge outage fails the gate

A run with zero results never passes.

Unknown gate keys are a configuration error, so a misspelt threshold fails loudly.

Model strings

<provider>/<model>: openai/… (any OpenAI-compatible base URL — OpenAI, OpenRouter, Ollama, vLLM; max_tokens_param forces the token parameter name), anthropic/… (official SDK, pip install "evalstone[anthropic]"), gemini/…. Keys are read from the environment and never stored. Unknown models are reported as unpriced, never guessed.

CenterCopy Anchor Link

TargetsCopy Anchor Link

GateCopy Anchor Link

Model stringsCopy Anchor Link

On this page

Center

Targets

Gate

Model strings