Alerts

AuditTrail's anomaly alert engine lets you define threshold rules over your agent telemetry and fire notifications when an observed metric crosses the threshold. Rules are tenant-scoped, evaluated on a background loop, and fan out through your configured webhooks.

How it works

Each alert rule names a built-in metric, a rolling window, a threshold, and a comparator. On its check_interval, the engine computes the metric over the last window_minutes of the caller's data and compares it to the threshold. When the comparison is true, an alert_events row is written and the event is delivered to your webhook destinations. Every rule and event is scoped by user_id, so cross-tenant access is impossible.

Rule fields

The supported metric identifiers are returned by GET /api/v1/alerts/metrics (the dashboard populates its dropdown from this), so create a rule only against a metric the engine actually computes — an unknown metric returns 400.

REST surface

routes/alerts.py — prefix /api/v1/alerts. All routes require a session cookie or an sk-at-… key (require_user_or_apikey).

Create a rule

POST /api/v1/alerts/rules
{
  "name": "Cost spike",
  "metric": "total_cost",
  "window_minutes": 30,
  "check_interval_minutes": 5,
  "threshold": 5.0,
  "comparator": "gte",
  "severity": "red",
  "enabled": true
}

Test before enabling

POST /api/v1/alerts/test/{rule_id} computes the metric over the rule's window right now and tells you whether it would fire — without writing an event. Use it to sanity-check a threshold before turning the rule on:

{
  "rule_id": "…",
  "metric": "total_cost",
  "observed_value": 3.21,
  "threshold": 5.0,
  "comparator": "gte",
  "would_fire": false,
  "context": { "...": "metric-specific detail" }
}

Events & delivery

Fired events are persisted to alert_events with the rule id, fired_at, the metric + observed value + threshold, severity, a metric-specific context blob, and a delivered flag. Delivery reuses the same outbound webhook machinery as constitutional violations (fan_out_violation), so an alert lands in Slack / PagerDuty / a signed HTTPS endpoint exactly like a governance violation. See the webhooks section of the API reference.

Data model