Datasets & evaluations

AuditTrail's offline evaluation surface lets you build datasets of test cases and run a built-in evaluator over them to score candidate outputs. This is distinct from the live, inline constitutional evaluations that run during ingest — those are documented under governance.

Datasets

A dataset is a named collection of items. Each item carries an input object, an optional reference_output (the gold answer, for evaluators that need it), and optional metadata. Datasets are versioned and can be archived. Everything is scoped by user_id.

routes/datasets.py — prefix /api/v1/datasets, require_user_or_apikey.

POST /api/v1/datasets/{dataset_id}/items
{
  "input": { "question": "What is 2 + 2?" },
  "reference_output": { "answer": "4" },
  "metadata": { "category": "arithmetic" }
}

Eval runs

An eval run executes one built-in evaluator over every item in a dataset and records a per-item score plus the run's mean score. The run is kicked off in a detached background task, so the POST returns immediately with the run id — poll the run (or its items) for progress.

routes/evals.py — prefix /api/v1/evals, require_user_or_apikey.

Starting a run

POST /api/v1/evals/runs
{
  "dataset_id": "…",
  "evaluator_id": "exact_match",
  "evaluator_config": {},
  "candidate_outputs": { "<dataset_item_id>": { "answer": "4" } }
}

The evaluator id must be one of the built-ins (GET /evals/evaluators) — an unknown id returns 400. For each item, the run scores the candidate output. The candidate comes from candidate_outputs[item_id] if you supply it, otherwise from the item's metadata.output. A run finishes complete when every item scored, or partial if any item errored; the mean score is computed over the items that produced a numeric score.

Run status shape

{
  "id": "…", "dataset_id": "…", "evaluator_id": "exact_match",
  "status": "complete", "total_items": 50, "completed_items": 50,
  "errored_items": 0, "mean_score": 0.94,
  "created_at": "…", "started_at": "…", "completed_at": "…"
}

Progress commits every 20 items, so a long run surfaces partial results via GET /evals/runs/{run_id}/items before it finishes.

Data model