Labeling outputs

Human labels are the ground truth (chapter 5, chapter 6). Standalone, Evalstone keeps them in a JSONL file you can version next to the dataset; attached to a center, the human check reads them from the center and the center's labeling queues produce them.

Grade a runCopy Anchor Link

evalstone run nightly --report r.json
evalstone label r.json --out evals/labels.jsonl

For every unique (sample, output) you see the input, the reference if there is one, and the output, then type p or f and an optional note. s skips, q quits; the next session resumes where you stopped. --scale 5 grades 1–5 (pass at 4; a 2-point scale passes only at 2, the judge's convention), --check tone records the grade for one rubric, --labeler names the grader. Outputs whose target errored are not offered: there is nothing to grade. The text is shown verbatim, whatever brackets or markup it contains.

A label is keyed by the sample id and a hash of the output, so when the answer changes the old grade does not apply. evalstone label always writes the local file (--out, default labels.jsonl), attached or not; see below for getting a file into the center.

Use labels as a checkCopy Anchor Link

checks:
  reviewed: { type: human, labels: evals/labels.jsonl }

A labelled output scores from its label; an unlabelled one is an error, never a pass. Keep human checks in their own suite for the slice that warrants it; they are the most expensive check there is.

labels: centerCopy Anchor Link

In an attached project labels: defaults to center (write it out, or leave it off): each (sample, output hash[, check]) is looked up in the center at run time, with --check-style per-rubric labels selected by the check's check: key. A missing label is the same error as above; a center that cannot be reached is an error score too, never a pass; and labels: center in a project with no center: block errors every result, so the gate fails loudly rather than quietly. A file path still works when attached — a frozen export, say.

Labels in and out of the centerCopy Anchor Link

The center's own labels come from labeling queues graded in the browser. A local labels.jsonl goes in with one request, and comes back out as the same format:

# import: upsert; later wins per (sample, output, check)
python3 -c 'import json,sys; print(json.dumps({"labels":[json.loads(l) for l in open("evals/labels.jsonl")]}))' \
  | curl -s -X POST "$API/projects/$PROJECT/labels" -H "Authorization: Bearer $EVALSTONE_API_KEY" \
         -H 'content-type: application/json' -d @-          # → {"upserted": 12}

# export: every live label as NDJSON — a drop-in labels.jsonl for the human check
curl -s "$API/projects/$PROJECT/labels/export" -H "Authorization: Bearer $EVALSTONE_API_KEY" > evals/labels.jsonl

Both accept the project's es_… ingest key or a signed-in member. The lines are the library's Label records (sample_id, output_hash, value, passed, notes, check, labeler, created_at, raw); a push is capped per request, so split a very large file. Endpoint details: center API.

Calibrate a judge against the labelsCopy Anchor Link

Once the same outputs carry a human label and a judge verdict, the center's Judges page computes agreement, Cohen's kappa, a pass/fail confusion matrix and the mean absolute error for that judge version, and Promote makes the calibrated model the project's override for that check. How to run it, and the labeler's keyboard shortcuts: the center, page by page.