Core observability signals for AI: request/response logs (what did users ask, what did the model respond), latency metrics (TTFT, tokens per second, total response time), cost tracking (tokens consumed, API spend), quality metrics (user feedback, automated quality scores), error rates (API failures, rate limits, content filter triggers), and safety metrics (refusal rates, flagged content, prompt injection attempts).
For complex AI applications (RAG pipelines, multi-agent systems), tracing follows a request through every step: the user query, the retrieval results, the prompt construction, the model call, the post-processing, and the final response. Each step is logged with inputs, outputs, latency, and cost. When something goes wrong, traces let you identify exactly where in the pipeline the failure occurred. LangSmith, Langfuse, and Braintrust provide LLM-specific tracing.
The hardest part of AI observability: automatically detecting when output quality degrades. Approaches include: LLM-as-judge (use a model to score outputs), embedding drift detection (if the distribution of outputs changes significantly, something may be wrong), user feedback signals (thumbs up/down, regeneration rates), and regression testing (periodically run a golden set of queries and compare outputs to baselines). No single approach catches everything — production systems use multiple signals.