Pause, inspect & resume

AuditTrail can act as a human-in-the-loop checkpoint for a running agent. The SDK announces a pause at a decision point; an operator inspects the captured state in the dashboard and resumes the run — optionally editing the state blob before it continues. The Edit + Resume path is feature-flagged and guarded by a schema-hash check so an operator can't silently corrupt agent state.

Lifecycle

A pause checkpoint moves through these statuses:

active ──▶ awaiting_edit ──▶ resumed
   │
   ├──▶ expired      (TTL elapsed)
   └──▶ abandoned    (heartbeat went stale)

• active — the SDK has paused and is polling/heartbeating for a resume.
• awaiting_edit — an operator has submitted an edit; the server has accepted the mutation and is about to flip to resumed.
• resumed — the operator (or the SDK's ack) released the pause.
• expired — the optional TTL passed before anyone resumed.
• abandoned — the SDK's heartbeat stopped (> 30s) while still active; a reaper on the online-eval loop flips it.

SDK side

The SDK opens a pause via audittrail.pause_checkpoint(label, state), which posts a checkpoint, heartbeats every ~5s, and long-polls for the resume. It computes a deterministic SHA-256 schema hash of the state snapshot so the resume path can detect a structurally different edited blob. Opt in with AuditTrailClient(pause_enabled=True).

The pause primitive is available in all four instrumentation SDKs: Python (pause_checkpoint), TypeScript (pause), Go (Pause) and Rust (pause). Each follows the same loop — register → heartbeat → block — and detects the resume the same way: /pauses/{trace_id}/active only ever serves non-terminal rows, so a resumed pause disappears from it; the SDK then reads the final state (resumed / expired / abandoned, plus any operator edit) from GET /pauses/{pause_id}.

Operator side

The dashboard renders a pause banner on the trace detail page. When the experimental_edit_resume flag is on (surfaced in the API response so the UI doesn't need a separate round-trip), an admin/editor can open the JSON editor, edit the captured state, and resume.

REST surface

routes/pauses.py — prefix /api/v1/pauses, require_user_or_apikey. Mutating routes are rate-limited 12/minute.

Resume safety cascade

POST /pauses/{pause_id}/resume accepts { reason?, edited_state_blob?, edit_note?, force? }. The guards apply in order:

When the edit is accepted, the server persists the new blob, records an audit_log row (pause.resume.edit) capturing the before/after hashes and the operator, flips through awaiting_edit → resumed, and broadcasts pause.requested → pause.ready → pause_resumed over the trace's WebSocket room. Resuming without an edited_state_blob is the legacy inspect-only path: it requires status active and just flips to resumed.

Feature flag & reaper

Data model

The pause_checkpoints table carries trace_id, span_id, parent_span_id, status, the state_snapshot + state_schema_hash, paused_at_step, ttl_until, heartbeat_at, and the resume metadata (resume_edit_json, resume_user_id, resume_reason, resumed_at).