The four monitoring layers
- Service: uptime, latency, errors, timeouts, and rate limits.
- Model: token use, model version, output format, and refusal behavior.
- Data: retrieval hits, stale sources, tool-call failures, and permission denials.
- Outcome: task success, corrections, escalations, user feedback, and cost per approved result.
Keep enough context to debug
For each request, store a trace ID, prompt version, model, route, retrieved source IDs, tool calls, timing, and outcome labels. Redact sensitive content and define retention before you turn logging on. The goal is to explain a bad answer without creating a second data leak.
Create alerts around user harm
A small latency increase may be annoying. A spike in unsupported claims, cross-tenant results, failed approvals, or unexpected cost is more urgent. Alert on thresholds tied to the product’s risk, not on every noisy model variation.