How it works

Evalstone measures quality in three layers. A judge — a model you choose, given a rubric and a small scale — scores what no string match can: whether an answer is good. Human labels are the ground truth that judge is calibrated against. Deterministic checks guard the contracts (schema, facts, length, safety, tool calls) exactly and for free, and run before the judges in every suite.

Standalone, Evalstone has no account, no proxy and no relay: when a check needs a model, the library calls the provider you configured, directly from the machine running the suite, with a key from your environment. Attached to a center, judge calls go through the center's model proxy instead, with the organization's key held server-side; the prompt target (the system under test) still runs locally on your providers:.

The judge: where quality is measuredCopy Anchor Link

A judge is a model, so it is reliable only when measured. Evalstone's judges use a small scale (a level and a reason, as JSON), hide the reference unless asked, judge both orders in a pairwise comparison, and treat a refusal, a network error or malformed JSON as an error, never a pass. Human labels (evalstone label, the human check) are what a judge gets measured against: agreement between the two is the number that says whether the judge can be trusted on your data. Standalone you get that number by running the human check and the judge on the same suite and reading the two columns; the agreement report itself (exact agreement, Cohen's kappa, a confusion matrix) is in the center, on the Judges page, where a calibrated judge can be promoted to the check's model. Chapter 6 of Evals 101, LLM-as-judge, is the background.

How a judge reaches a providerCopy Anchor Link

A model is named as provider/model, either per check (model:) or once for the project:

judges:
  default_model: anthropic/claude-haiku-5-5
providers:
  anthropic: { api_key_env: ANTHROPIC_API_KEY }
  openai:    { api_key_env: OPENAI_API_KEY }
  gemini:    { api_key_env: GEMINI_API_KEY }

Three adapters ship with the library:

OpenRouter is not built in and not required; it is one base_url the openai adapter can point at, like Ollama, vLLM, Groq or Together. One OpenRouter entry gives you every model it hosts, and switching the judge is one line: judges.default_model for the project, or model: on a single check (a switched judge re-judges, since verdicts are cached per model).

providers:
  openai: { base_url: https://openrouter.ai/api/v1, api_key_env: OPENROUTER_API_KEY }
judges:
  default_model: openai/anthropic/claude-haiku-5-5   # OpenRouter's model id after the provider key

providers:
  openai: { base_url: http://localhost:11434/v1 }     # Ollama: no key
judges:
  default_model: openai/llama3.1

Rules that hold for every adapter:

• Keys come from environment variables by default. api_key_env: names which one. A provider entry also accepts an inline api_key: for programmatic or test use; it is excluded from every dump and report, but a key in a committed evalstone.yaml is still a committed key, so prefer the environment. The Anthropic adapter also lets the official SDK resolve a key from its own login profile when neither is set.
• Verdicts are cached by content for the life of the process. The cache key is the judge's version (rubric, scale, model) plus the exact prompt, so within one run — or one MCP server session, which keeps its cache across calls — an unchanged output on an unchanged judge is never sent twice. Each evalstone run is a new process with an empty cache; a cache shared across runs and machines is what an attached center provides.
• Calls are priced from a built-in table of USD per million tokens, which today lists Claude models. A model that is not in it is reported unpriced and counts as zero toward max_cost_usd, so add your models under prices: before relying on that cap.
• No key, a network error, a refusal, malformed JSON: an error, never a pass. The gate's max_error_rate defaults to 0, so a judge outage fails the gate instead of hiding inside a pass rate.
• Thresholds live outside the cache. A judge caches the model's raw level and reason; pass_level / threshold are applied on read, so changing the bar within a process does not re-spend.

Attached to a centerCopy Anchor Link

With a center: block the judges' registry is the center's model proxy: the library sends the judge prompt to POST /model/complete with an ingest key, the center resolves the provider from the organization's saved keys and dials it. judges: is then ignored, and providers: no longer serves the judges — it still serves a prompt target, the system under test. The model a judge uses follows one precedence: --judge-model on the command line, then the project's per-check override from the center's Judges page (what Promote writes), then the check's own model:, then the project's default in the center; the full list, with what differs standalone, is in the config reference.

Verdicts are read from and written to the center's judge cache under the resolved model, so one verdict serves every machine and a changed override re-judges. One caveat: attached, --judge-model is not validated before the run — a typo'd provider is rejected by the center at the first judge call and surfaces as an errored score, which the error-rate gate still catches.

The guardrails: what runs with no model at allCopy Anchor Link

These do not judge quality; they catch the contract failures that break production, exactly and for free. Deterministic and trajectory checks are sorted before the judges in every suite, and so are safety rules and human labels (everything that costs nothing). By default every check still runs for every sample, so a judge is spent on an output whether or not a contract check failed it; short_circuit: true on a suite skips the judges for a sample a contract check (deterministic, safety, trajectory; not a human label) already failed — recorded as a skipped failure, never an error — so no paid call is spent on it. A skipped judge counts as a failure, so re-take your baseline when you turn it on. The quickstart's first run is exactly this: recorded outputs, deterministic checks, a gate. Nothing is called, nothing costs money, and the result is the same every time.

A callable or http target runs your code, which may well call a model; that is your feature under test, not Evalstone.

Where the data goesCopy Anchor Link

Without a center: nowhere but your provider. A run is a JSON file you keep, commit or upload as a CI artifact. With a center: block the run is also pushed to the center — hosted or self-hosted — which stores it in PostgreSQL next to your datasets, labels and monitors; the library stays usable without it.