Natural Language Explanations

The "Why this tool?" card inside the Sankey view on a trace detail page translates SHAP feature-importance scores into a plain-English paragraph. If you have an LLM provider key configured, the paragraph is generated by a small judge model; otherwise a deterministic template fallback keeps the surface useful.

Why a second surface (SHAP isn't enough)?

SHAP scores are dense: feature: query_specificity · weight: 0.41 tells a data scientist a lot and a product manager nothing. The NL explanation is a two-sentence synthesis designed to onboard non-technical reviewers without hiding the underlying numbers — the SHAP bars stay visible below the card.

How it renders

On /traces/[id] → Sankey tab, after the ablation has run:

┌────────────────────────────────────────────────────────┐
│  ✨ Why this tool?                 [ Regenerate ]     │
│  Natural-language summary of the SHAP attributions     │
│  for web_search                                        │
│                                                        │
│  The agent selected `web_search` primarily because    │
│  `query_specificity` contributed 41% of the decision  │
│  weight. Secondary factors were `recency_filter` and  │
│  `source_quality`.                                     │
│                                                        │
│  Powered by gpt-4o-mini.                               │
└────────────────────────────────────────────────────────┘

Footer text shows either "Powered by judge_model" or "Template fallback (no provider key)". The Regenerate button re-runs the call without touching the underlying SHAP data.

Caller pattern

The endpoint is stateless — no span_id parameter. The frontend first loads SHAP attributions for the selected span (via the existing ablation hooks), then passes those attributions in:

  1. useAblationShap(spanId) → returns { features[], tool_predicted, … }
  2. Map features → [{ phrase, score }]
  3. useNlExplanation() mutate with { agent_action: tool_predicted, attributions, prompt, judge_model }
  4. Render the returned { explanation, source } paragraph

This way, any surface that has attributions (counterfactual panels, assistant tiles, regression reports) can re-use the same NLG route without duplicating SHAP fetches.

Endpoint

POST /api/v1/xai/explain-attributions

json
// Request
{
  "agent_action": "web_search",
  "attributions": [
    { "phrase": "query_specificity", "score": 0.41 },
    { "phrase": "recency_filter",    "score": 0.22 },
    { "phrase": "source_quality",    "score": 0.15 }
  ],
  "prompt": "latest AI safety papers 2026",
  "judge_model": "gpt-4o-mini"
}
 
// Response
{
  "explanation": "The agent selected `web_search` primarily because query_specificity contributed 41% of the decision weight…",
  "source": "llm"
}

source is "llm" when a provider key is configured (AUDITTRAIL_NLG_OPENAI_KEY / AUDITTRAIL_NLG_ANTHROPIC_KEY), otherwise "template".

User-scoped (bearer or sk-at-*). Rate limited at 20/minute per client because it calls an external LLM.

UI mount point

The <NlExplanationCard /> component mounts at the top of the InlineSankeyView panel on /traces/[id], above the SHAP / Ablation / Tool-Probabilities grid. It also renders when the standalone /sankey?traceId=… page has SHAP data available.

Shipped in: prod-v2.6.4 — backend nlg.py was in place since V1.3; the UI card lands this sprint.