17 — LLM Observability: Traces, Cost, Latency, and What to Log

A log line and a status code used to carry me a long way — until I was debugging a system where the same input produced different outputs across calls. The model that covers both worlds: LLM observability is traditional observability — traces, logs, metrics — plus one new axis, output quality, and the traces are what let you debug the non-determinism. [1] A traditional service is mostly deterministic: given the same input, it produces the same output, so a log line and a status code carry you a long way. An LLM system isn't. The same input can produce different outputs across calls, across model versions, across days. That non-determinism is why traces — the full record of what happened during a request — stop being optional and become the primary debugging tool.

The framing that finally landed is a layering of what to watch. At the base, the same signals as any service: errors, latency, cost. On top of those, two things unique to LLM systems: the full trace of every step (prompts, model calls, tool calls, retrieved context, responses), and a quality signal that says whether the output was actually good. Without the trace, I can't reconstruct what made a weird output happen. Without the quality signal, I can't tell that a system is slowly degrading until users complain.

Tracing and logging: the reconstruct-the-request view

Tracing records the full lifecycle of a request through the AI system — from the initial user input, through any intermediate LLM calls, tool uses, and retrieval steps, all the way to the final response [2]. Logging captures individual events like errors, latency spikes, or unexpected outputs. Together they let me reconstruct exactly what happened during any given interaction.

For a deterministic service, a log line is usually enough — the same input produced this output, here's the timestamp. For an LLM system, that's nowhere near enough. A weird output might have come from a retrieved chunk that was slightly off, a tool that returned stale data, or the model sampling an unlikely token — and none of those show up in a request log. The trace is the only thing that lets me answer "why did it produce _this_?" Without it, debugging agents and multi-step pipelines is mostly guesswork.

Cost and latency: the budget signals

Cost and latency monitoring tracks token usage, the resulting financial cost, and response times across the system [3]. Without this visibility, production costs compound silently — especially with large reasoning models that charge significantly per token. The classic failure mode: a feature works great in testing, ships, and the bill is ten times what anyone budgeted, because no one watched the token counts.

The discipline here is to track cost and latency alongside quality, because they trade off. Routing simpler queries to a cheaper model, caching common responses, or pruning the context window all reduce cost and latency, often without hurting quality. Without measurement I can't make those tradeoffs deliberately; with it, "route this query type to the small model" becomes a data-backed decision rather than a guess.

Constraining inputs and outputs: prevention as observability's partner

Observability tells me what happened; constraints shape what _can_ happen. Constraining inputs and outputs means setting rules on both sides — input validation ensures only clean, well-formed data reaches the model, and output constraints limit length, enforce format, or filter harmful content [4]. Structured outputs (covered earlier) are the strongest output constraint: a schema-constrained response literally cannot be a free-form rant. From the observability side, I watch for inputs that violated my constraints (a sign someone is probing the system) and outputs that triggered filters (a sign the model is drifting).

Production monitoring: where the real edge cases live

Production monitoring is the continuous observation of the system once it's handling real user traffic [5]. The honest reason it deserves its own name: development and staging never surface the edge cases production does. Real users send inputs I didn't anticipate, in combinations I didn't test, at volumes that stress every assumption. Production monitoring tracks quality metrics, error rates, and behavioral changes over time, so regressions and anomalies get caught before they affect many users.

The piece I underrated for too long: behavioral change over time. Models get updated by vendors, prompts get edited, retrieval indices grow — and any of those can silently degrade quality. A dashboard that shows "quality score this week versus last week" is what turns a slow regression into something I catch on Tuesday instead of in next month's user complaints.

The adjacent practices: moderation, adversarial testing, knowing your customer

A few related practices round out the picture. Content moderation APIs automatically analyze text, images, video, and audio for harmful or policy-violating content, letting me filter problematic user-generated or model-generated output [6]. Adversarial testing intentionally exposes the model to deceptive, perturbed, or crafted inputs to find vulnerabilities before attackers do [7]. Robust prompt engineering minimizes ambiguity and anticipates misinterpretation, reducing the surface area for failure [8]. And knowing your customer — deeply understanding users' needs and defining the tool's functionality and boundaries — ensures the feature is built for its actual use and protected against misuse it wasn't designed for [9]. These aren't observability per se, but they're the practices that make the observability signal actionable: moderation gives me a flag to log, adversarial testing gives me the inputs to watch for, and knowing the customer tells me which behaviors actually matter.

How I use this

Observability is now a first-class part of how I build, not an afterthought. Every LLM feature ships with tracing on from day one — I log the full prompt, the retrieved context, every tool call and its result, and the final response, tied together by a request ID. I add cost and latency dashboards before launch, because a feature whose bill I can't see is a feature that will surprise me. I add a quality signal — even a simple thumbs-up/thumbs-down or an LLM-as-judge score on a sample — because without it I'm blind to slow degradation. And I review the traces of weird outputs personally, on a cadence, because that's where the real learning happens: every strange response is a hint about a retrieval gap, a prompt weakness, or a tool bug that no aggregate metric would have surfaced. The discipline is to treat the trace as the unit of debugging and quality as a first-class metric, because in a non-deterministic system, those two are what keep me honest.

References

[1] IBM, "What is LLM observability?," 2024. [Online]. Available: https://www.ibm.com/think/topics/llm-observability

[2] Weights & Biases, "A guide to LLM debugging, tracing, and monitoring," 2024. [Online]. Available: https://wandb.ai/onlineinference/genai-research/reports/A-guide-to-LLM-debugging-tracing-and-monitoring--VmlldzoxMzk1MjAyOQ

[3] Morphllm, "LLM Cost Optimization: 5 Levers That Cut API Spend 70-85%," 2024. [Online]. Available: https://www.morphllm.com/llm-cost-optimization

[4] OpenAI, "Introducing Structured Outputs in the API," 2024. [Online]. Available: https://openai.com/index/introducing-structured-outputs-in-the-api/

[5] Swept.ai, "LLM Observability: The Complete Guide to Monitoring LLMs in Production," 2024. [Online]. Available: https://www.swept.ai/post/llm-observability-complete-guide

[6] OpenAI, "Moderation API," 2024. [Online]. Available: https://platform.openai.com/docs/guides/moderation

[7] Google, "Adversarial Testing for Generative AI," 2024. [Online]. Available: https://developers.google.com/machine-learning/resources/adv-testing

[8] AIM Research, "Building Robust Prompt Engineering Capability," 2024. [Online]. Available: https://aimresearch.co/product/building-robust-prompt-engineering-capability

[9] Learn Prompting, "Assigning Roles," 2024. [Online]. Available: https://learnprompting.org/docs/basics/roles