Center API

The center is a FastAPI service; the full OpenAPI document with every field lives at api.evalstone.com/docs (a self-hosted center serves the same at /docs). This page is the short list of what you call from outside the web app. Every path below is relative to the API base URL (https://api.evalstone.com); {project_id} is the project's uuid, {organization_id} the organization's.

AuthenticationCopy Anchor Link

Two credentials exist:

Both go in Authorization: Bearer <value>. Ingest keys are minted on the project's Settings page by an organization admin (POST /projects/{project_id}/ingest-keys with {"name": "ci"}); the plaintext is returned once, the center keeps a hash and a prefix. Revoke with DELETE /projects/{project_id}/ingest-keys/{key_id}. A key is bound to one project: calling another project's path with it is a 404, as if the project did not exist.

curl -H "Authorization: Bearer $EVALSTONE_API_KEY" https://api.evalstone.com/ingest/whoami
# → {"id": "…", "organization_id": "…", "name": "support-bot", "slug": "support-bot", …}

GET /ingest/whoami is the smoke test: it answers with the project the key belongs to, or 401.

Runs and baselinesCopy Anchor Link

The web app's runs page has a "Set as baseline" button that calls the PUT.

Judge settings and the model proxyCopy Anchor Link

LabelsCopy Anchor Link

The center stores one label per (sample, output hash, check); the latest write wins. Labels arrive from the web app's labeling queues or from a local labels.jsonl.

# push what `evalstone label` wrote
jq -s '{labels: .}' labels.jsonl | curl -X POST -H "Authorization: Bearer $EVALSTONE_API_KEY" \
  -H 'Content-Type: application/json' -d @- https://api.evalstone.com/projects/$PROJECT/labels

# pull everything graded in the center
curl -H "Authorization: Bearer $EVALSTONE_API_KEY" \
  https://api.evalstone.com/projects/$PROJECT/labels/export > evals/labels.jsonl

CalibrationsCopy Anchor Link

A calibration compares a judge's verdicts with the human labels for the same (sample, output) pairs and reports agreement, Cohen's kappa, mean absolute error and a confusion table.

Member role or above. The Judges page runs calibrations and, from a calibration, lets an admin promote the judge version into the project's per-check override (the PUT /projects/{project_id}/judge-settings above).

Traces, monitors and alertsCopy Anchor Link

Production traffic goes in with the ingest key; a monitor scores a sample of it with checks from the organization's catalog and opens an alert when a windowed metric drops below a threshold.

curl -X POST -H "Authorization: Bearer $EVALSTONE_API_KEY" -H 'Content-Type: application/json' \
  https://api.evalstone.com/projects/$PROJECT/traces -d '{
  "traces": [{
    "id": "req-8f3a",
    "input": {"question": "Can I return a gift?"},
    "output": "Yes — within 30 days with the receipt.",
    "metadata": {"route": "/answer"}, "tags": ["returns"],
    "model": "anthropic/claude-haiku-5-5", "latency_ms": 812, "cost_usd": 0.0004,
    "occurred_at": "2026-10-14T09:12:00Z"
  }]
}'
# → {"stored": 1}

id is your own request id: a retry carrying the same id is stored once.

Every minute the center scores the unscored traces whose stable sample bucket falls under sample_rate (deterministic checks in-process, judges through the project's judge settings), evaluates the policy over the last window_minutes once at least min_n traces are scored, opens an alert when the metric is below threshold, and resolves it when the window is back at or above. A new alert POSTs this JSON to webhook_url (an http(s) URL with a resolvable public host and no embedded credentials — checked at save time); delivery failures land in the alert's notify_error:

{
  "event": "evalstone.alert.opened",
  "alert_id": "…", "monitor": {"id": "…", "name": "prod answers"}, "project_id": "…",
  "metric": "pass_rate", "value": 0.83, "threshold": 0.9,
  "n": 24, "window_minutes": 1440, "opened_at": "2026-10-14T09:13:00Z"
}

Check catalogCopy Anchor Link

Organization-wide check definitions (the same type: + kwargs as a checks: block) that monitors reference by key and the web app edits with versions.

Catalog routes take a user token (members read, admins write), not an ingest key.