Checks catalog
Every built-in check, what it needs from the sample, and what it returns.
Every check returns a Score: value in 0..1, passed, a reason, cost and
latency, and error when it could not score. An error is never a pass.
In Python the same factories live under evalstone.checks; in YAML the
type: is the name below.
Deterministic (free, repeatable)
| Type | Reads | Passes when |
|---|---|---|
exact | expected: literal, else the sample's expected (or path: into it); case_sensitive, strip | output equals the reference (text or JSON equality) |
contains | expected.must_include (or path: / needles:), require_all, case_sensitive | every (or any) needle appears; value = fraction found |
regex | pattern, flags (an int: re flag bits, e.g. 2 for IGNORECASE), must_match | the pattern matches (or does not) |
json_schema | schema (JSON Schema dict — draft from $schema — or a pydantic model) | the output is valid JSON and conforms; malformed schemas fail at load |
numeric_tolerance | expected: literal, else the sample's expected (or path:); output_path, abs_tol, rel_tol | the output number is within tolerance |
length | max_chars, max_words, min_chars | within limits |
python | (Python only) any fn(sample, output[, ctx]) | whatever it returns (Score, bool, or 0..1) |
A missing or null reference is an error.
LLM-as-judge
| Type | Reads | Notes |
|---|---|---|
judge | rubric, scale (default 5), pass_level (default: second-highest level; the top level on a 1–2 scale) or threshold (0..1), model, include_reference (default false), max_tokens (default 1024) | value = (level−1)/(scale−1); verdicts cached by judge version + prompt — within one process standalone, across runs and machines when the project is attached to a center — so unchanged outputs cost nothing again |
RAG (judge-backed; the sample input carries question and context)
| Type | Reads | Grades |
|---|---|---|
rag.faithfulness | context, model, threshold | every claim in the answer is supported by the context |
rag.context_relevance | question, context, model, threshold | the context can answer the question (the answer is ignored) |
rag.answer_relevance | question, model, threshold | the answer addresses the question (accuracy is ignored) |
question: and context: name the input fields; threshold is 0..1. These
take only the keys listed — not the judge's scale, pass_level or
include_reference — because each ships its own fixed rubric and scale.
Safety (rule-based)
| Type | Reads | Passes when |
|---|---|---|
safety.refusal | expected: literal (bool), else the sample's expected[path] with path default should_refuse; neither set = must not refuse | the output refuses exactly when it should |
safety.no_pii | kinds ⊆ email, phone, card (Luhn-valid), ssn, ipv4 | none found |
safety.forbidden | phrases, case_sensitive | none of the phrases appear |
Heuristics, not a DLP engine; keep safety checks in their own suite.
Pairwise comparison
| Type | Reads | Value |
|---|---|---|
comparison.pairwise | rubric, baseline (a target spec or {sample_id: output}), model, both_orders (default true), max_tokens (default 1024) | mean over judged orders of 1 (candidate preferred) / 0.5 (tie) / 0; passes at 0.5 |
Both orders are judged by default to cancel position bias.
Agent trajectory (deterministic; output.value["steps"] = [{"tool", "args"}, …])
Every step-reading check takes steps: (default "steps"), the key under
output.value (or output.metadata) holding the trajectory.
| Type | Reads | Passes when |
|---|---|---|
trajectory.tool_called | name, min_times (default 1), max_times, args (Python predicate) | call count within bounds — with the default an empty trajectory fails |
trajectory.tool_order | before, after | every after is preceded by a before |
trajectory.max_steps | n | at most n steps |
trajectory.no_repeated_calls | max_repeats | no identical (tool, args) call repeats — loop detection |
trajectory.finished | path (default answer) | output.value[path] is non-empty (reads no steps; a missing answer is a plain fail) |
For the four step-reading checks, no recorded trajectory at all (no steps
list on the output) is an error, so a broken recorder trips
max_error_rate; an empty trajectory is judged like any other (it passes
tool_order, max_steps and no_repeated_calls, and fails tool_called
unless min_times: 0).
Human
| Type | Reads | Passes when |
|---|---|---|
human | labels (a JSONL path from evalstone label, or center), check | a label exists for this sample and this exact output, and it passed |
labels: center looks the label up in the center at run time; it is the default
for an attached project (unattached, the
default is labels.jsonl next to the config). A center hiccup or labels: center
in an unattached run is an error score, never a pass, so it counts
toward max_error_rate.
Your own
from evalstone import check
@check("no_apology")
def no_apology(sample, output):
return "sorry" not in output.text.lower()List the module under imports: and use the key in a suite. Re-importing the
module replaces the check; a different function under the same key is a
duplicate error.