A production failure without observability
A support assistant starts returning weak answers. Without a trace, the team may blame the model. With one, they can see that a document sync stopped three days earlier, retrieval returned empty context, and a fallback prompt encouraged the model to answer anyway.
What to observe
- Request, user, tenant, and permission context.
- Model, version, settings, and token usage.
- Retrieved documents and tool calls.
- Latency, retries, errors, and cost.
- Quality labels, corrections, and user feedback.
Observability is not surveillance
Logs should be designed around the least data needed to debug the system. Redact sensitive content, restrict access, define retention, and let operators inspect the decision path without creating a second privacy problem.