Documentation

AuditTrail Architecture Overview

System Diagram

                          +------------------+
                          |   Browser / UI   |
                          |  (Next.js App)   |
                          +--------+---------+
                                   |
                          HTTP / WebSocket
                                   |
                          +--------+---------+
                          |      Caddy       |
                          | (Reverse Proxy)  |
                          +---+---------+----+
                              |         |
                 /api/, /ws/  |         |  / (everything else)
                              |         |
                    +---------+--+   +--+---------+
                    |  FastAPI   |   |  Next.js   |
                    |  Backend   |   |  Frontend  |
                    |  :8000     |   |  :3000     |
                    +-----+------+   +------------+
                          |
          +---------------+---------------+
          |               |               |
   +------+------+ +------+------+ +------+------+
   |   Trace     | | Causal      | | Constitu-   |
   |   Collector | | Attribution | | tional      |
   |   + DAG     | | Engine      | | Governor    |
   |   Builder   | | (Ablation   | | (Rule       |
   |             | |  + SHAP)    | |  Engine)    |
   +------+------+ +------+------+ +------+------+
          |               |               |
          +-------+-------+-------+-------+
                  |               |
           +------+------+ +------+------+
           |   SQLite    | |  Rule YAML  |
           |  (WAL mode) | |  Files      |
           +-------------+ +-------------+

Component Descriptions

Frontend (Next.js 16 + shadcn/ui)

The frontend is a Next.js 16 App Router application using React 19. Component primitives are sourced from @base-ui/react via the shadcn base-nova style (not @radix-ui/* directly). Animation uses motion v12 (the rebranded Framer Motion package, imported as motion/react). The primary dashboard surfaces:

  1. Overview -- KPI cards, recent traces, sparklines.
  2. Traces -- Paginated, filterable list with drill-down to a detail page whose tabs are Spans, DAG (interactive React Flow decision tree, nodes color-coded by span type/status), Timeline, Sankey (causal attribution — hand-rolled SVG layout; flow width ∝ attribution strength), Counterfactuals, and SAE (mechanistic features, gated on self-host extras).
  3. Fleet -- Rolling per-agent health (traces/min, error rate, p95, violations) over a configurable window.
  4. Analytics -- Recharts dashboards for cost, latency, tool usage, and the Fleet WHY tab (surrogate global feature importances).
  5. Compliance -- Real-time constitutional status: pass/amber/red donut, policy-group tiles, rule table, EU-AI-Act module.
  6. Reports -- Audit report generation (PDF/JSON) with period selection.
  7. Deployments / Run Status / Prompts / Alerts -- the control plane: 3-tier action queue with evidence panels, live runtime controls with TTLs, prompt canaries, anomaly alert rules.
  8. Assistant -- BYOK operations chatbot with generative-UI tiles.
  9. Settings + Admin console -- tabbed settings (rules, gateway, keys, runner, …) and the superadmin console (users, audit log, tenants, SQLite health at /admin/database).

State management uses Zustand for ephemeral UI state (sidebar collapse, command palette, websocket buffers). Server state lives in TanStack Query with a 60-second stale time for analytics and dashboard endpoints.

Real-time updates are delivered via WebSocket connection to the backend (/ws/traces/{trace_id} for a single trace, /ws/live for the global feed). The envelope key is event (not type). Each /ws/live connection is tenant-scoped — viewers only receive events for traces they own; admins receive everything.

Backend (FastAPI)

The backend is a Python FastAPI application providing REST endpoints and WebSocket connections. It has four major subsystems:

Trace Collector + DAG Builder

The collector ingests trace and span data from instrumented agents via the REST API. It stores raw trace data in the database and reconstructs the execution DAG on demand by traversing the span parent-child tree.

The middleware hooks into LangGraph's callback system to capture full state at every node transition -- inputs, outputs, model parameters, token counts, timing, and error state. Middleware is async and non-blocking to stay within the <100ms latency budget.

Causal Attribution Engine

The ablation engine implements prompt-level causal attribution:

  1. Segmentation -- Split the user prompt into meaningful phrases.
  2. Ablation -- For each phrase, mask it and re-run the agent to measure which tool selections change.
  3. Averaging -- Run each ablation 3x to reduce noise from LLM non-determinism. Report confidence intervals.
  4. SHAP -- Train a surrogate model on ablation results and compute SHAP values for fine-grained feature importance.
  5. Sankey Construction -- Build the Sankey diagram data structure mapping phrases to reasoning steps to tool calls.

Ablation is opt-in and uses cheaper models (configurable, defaults to a small model) for re-runs. Results are cached by prompt hash so repeated analyses are instant.

Constitutional Governor

The governor evaluates every span against a set of rules defined in YAML files. Rules specify:

  • Target -- Which span types to evaluate (tool, llm, chain, or all).
  • Condition -- A Python expression evaluated against the span data.
  • Thresholds -- Amber (approaching boundary, default 80%) and red (violation) levels.

The governor's key insight is boundary detection: it flags actions that APPROACH a violation (amber) even if they don't cross it. This "almost violated" signal is more informative than actual violations for proactive governance.

Evaluation results are stored as ConstitutionalEvaluation records and streamed to the frontend via WebSocket for real-time toast notifications.

Report Generator

Generates PDF audit reports using ReportLab or WeasyPrint. Reports include trace summaries, constitutional evaluation results, analytics charts, and compliance recommendations for a specified time period.

Database

SQLite in WAL mode is the database in both development and production (via aiosqlite for async support), tuned with busy_timeout=5000 and synchronous=NORMAL. A superadmin health page at /admin/database reports the live file/WAL sizes, PRAGMA counters, and per-table row counts. PostgreSQL (via asyncpg) is a supported future cutover target — the Alembic migration chain reproduces the full schema for that purpose, and a migration-parity test pins alembic upgrade head against the ORM metadata.

Core tables (the spine — the full schema is ~50 tables):

  • traces -- Top-level agent execution records
  • spans -- Individual execution steps within traces
  • tool_calls -- Tool invocations linked to spans
  • constitutional_evaluations -- Rule evaluation results
  • ablation_results -- Causal attribution analysis outputs
  • rules -- Legacy rule metadata (the governor loads and evaluates rules from the YAML files below, not this table)
  • agents -- Registered agent metadata
  • api_keys -- Authentication keys for trace ingestion
  • users -- Dashboard user accounts
  • reports -- Generated audit report metadata
  • settings -- System configuration key-value pairs

Constitutional Rules (YAML)

Rules are defined in YAML files in the rules/ directory (one rule per file). The backend loads and validates them at startup using Pydantic. The condition is a structured field/operator object — never a Python expression string — and each threshold carries its own value + message:

yaml
rule:
  id: cost-guard-001
  name: "Per-step cost limit"
  description: "Flag spans where estimated LLM cost exceeds thresholds"
  version: "1.0"
  target:
    span_type: llm
  condition:
    field: "estimated_cost"
    operator: "gte"
  thresholds:
    amber:
      value: 0.05
      message: "LLM call cost exceeds $0.05 — review for optimization"
    red:
      value: 0.25
      message: "VIOLATION: LLM call cost exceeds $0.25 — requires justification"
  enabled: true
  policy_group: "cost"

Data Flow

Trace Ingestion Flow

Agent (LangGraph)
  -> Middleware captures span data (async, &lt;100ms)
  -> POST /api/v1/ingest/traces
  -> Collector validates and stores in DB
  -> WebSocket broadcasts trace.started / span.created events
  -> Frontend updates DAG and trace list in real-time

Causal Attribution Flow

User clicks "Analyze" on a span in the UI
  -> POST /api/v1/ablation/run (with confirmed: true after the estimate)
  -> Backend estimates cost, returns estimate
  -> User confirms
  -> Ablation engine segments prompt, runs ablation passes
  -> WebSocket broadcasts ablation.progress events
  -> On completion, SHAP values computed, Sankey data built
  -> Frontend renders interactive Sankey diagram

Constitutional Evaluation Flow

Span ingested by collector
  -> Governor evaluates span against all enabled rules
  -> Evaluation results stored as ConstitutionalEvaluation records
  -> If severity >= amber, WebSocket broadcasts constitutional.alert
  -> Frontend shows toast notification and updates compliance dashboard

API Communication Pattern

The frontend communicates with the backend via two channels:

  1. REST API (HTTP) -- All CRUD operations, queries, report generation, ablation triggers. Endpoints are versioned under /api/v1/. Request/response bodies use JSON with Pydantic validation on the backend and Zod validation on the frontend.

  2. WebSocket (/api/v1/ws/*) -- Real-time event streaming. The frontend opens a single persistent WebSocket connection on page load. Events are JSON objects whose envelope key is event (not type — see the note in the Frontend section), with the event data alongside. Event types include trace lifecycle, span updates, constitutional alerts, and ablation progress.

Database Choice Rationale

SQLite (WAL mode) is the database in development and production. It requires zero infrastructure setup — the database is a single file created automatically — which eliminates the "install PostgreSQL first" barrier for contributors, and WAL mode plus a single-writer FastAPI process comfortably serves the reference deployment's scale. The /admin/database superadmin page exists precisely to keep an eye on file growth, WAL checkpointing, and per-table row counts on a production SQLite install.

PostgreSQL (via asyncpg) remains the documented cutover target for installs that outgrow a single writer:

  • Concurrent write support (SQLite serializes writers)
  • JSONB columns for efficient metadata queries
  • Connection pooling for high-throughput ingestion

The application uses SQLAlchemy with async drivers (aiosqlite / asyncpg), so the cutover path is: point AUDITTRAIL_DATABASE_URL at Postgres and reach full schema via the Alembic chain (alembic upgrade head, with DATABASE_URL set for the migration runner). A committed migration-parity test keeps the chain byte-equivalent to the ORM metadata.