In an LLM feature, an unspecified behavior produces a confident, fluent, wrong answer.
Reading the Trace, How to Diagnose an AI Failure After It Happens
When an AI agent produces a bad output, 'the model hallucinated' is almost never the complete explanation. This article shows PMs how to read LLM trace logs to reconstruct the actual chain of decisions that caused the failure and why that chain almost always starts with a spec gap.
What it is: How to use LLM trace logs to reconstruct why an AI agent produced a bad output, and what a PM's role in that investigation actually is.
The Ticket That Explains Nothing
The bug report lands in your queue: "The AI gave wrong information about our pricing." Someone screenshots the output. Someone else pings the engineering lead. Within an hour, a ticket exists with the label: model hallucination.
The ticket is not wrong. The model did produce something false. But calling it a hallucination and closing the loop there is the equivalent of a pilot writing "plane fell" in the incident log.
The model's output is the end of a chain. Every link in that chain was shaped by a decision, what context to retrieve, how to structure the prompt, what constraints to specify, what to leave unspecified. Most of those decisions were made by a PM or a PM-adjacent person, often informally, often months earlier.
Trace analysis is how you find those decisions again.
flowchart TD
A[Prompt] --> B[Context Window]
B --> C[Tool Calls]
C --> D[Reasoning Steps]
D --> E[Final Output]
A --> A1{Are instructions clear\nand scope bounded?}
B --> B1{Is retrieved context\nrelevant and current?}
C --> C1{Did tool calls match\nproduct intent?}
D --> D1{Did reasoning drift\nor loop?}
E --> E1{Does output match\nspec constraints?}What a Trace Actually Contains
When a language model processes a request, modern observability tooling captures a record of everything that happened between the user's input and the final response. That record is a trace.
A complete trace contains several distinct layers, and not all of them belong to the same reader.
The prompt sent to the model, the full text, including system instructions and any injected context. This is the PM's primary reading zone. If the instructions are vague, contradictory, or silent on edge cases, that is visible here.
The context window contents, what documents, data, or retrieved chunks were included alongside the prompt. This is where retrieval-augmented systems reveal their failure modes. The PM reads this to ask: what did the model actually know at the time? Was the retrieved content relevant, current, and scoped to the task?
Tool calls and intermediate outputs, if the agent called external tools (a search index, a database, a calculator), the trace records each call, its inputs, and its response. Engineering reads this for correctness of the tool integration. The PM reads this for intent: did the tool call reflect what the product was supposed to do?
Intermediate reasoning steps, in chain-of-thought systems or agents with explicit planning steps, the trace shows the model's working. This is where you find reasoning loops, premature conclusions, or scope drift before the final answer was generated.
The final output with token metadata, what the user saw, plus latency and token count. Mostly engineering territory, though token counts near the context limit are a signal a PM should care about.
Most PMs never read traces. They read the final output, form a hypothesis, and hand it off. That is the gap this article is trying to close.
Surface-Level Bug Report vs. Trace Analysis
The difference between filing a ticket and reading a trace is not a difference in effort, it is a difference in what you can actually fix.
| Dimension | Surface-Level Bug Report | Trace Analysis |
|---|---|---|
| What you know | The output was wrong | Which input layer caused it |
| Fixability | Unclear, model may ship again with same failure | Targeted, fix the prompt, context, or retrieval spec |
| Time to resolution | Days to weeks (engineering reproduces, investigates) | Hours (the record already exists) |
| Who owns it | Defaulted to engineering | Split: PM owns prompt/spec layer, engineering owns execution layer |
| Recurrence risk | High, root cause unknown | Low, root cause traceable and preventable |
| What gets written down | A ticket with symptoms | A spec change with a rationale |
The column on the right is not faster because PMs are smarter. It is faster because the evidence is already there. A trace is a deterministic replay of what happened. The only question is whether someone reads it.
The Pattern That Emerges From Systems Like Copilot
When teams read traces from code assistant systems, tools in the same category as GitHub Copilot, a specific failure pattern appears repeatedly.
The assistant suggests an API method. The method was valid at some point. It has since been deprecated and removed from the current version of the library. The user follows the suggestion. The build breaks.
The surface report: the model hallucinated valid code.
The trace tells a different story. The retrieved context, the documentation or code examples injected into the prompt before the model generated its suggestion, came from an index that was last updated well before the deprecation occurred. The model did not invent the method. The model accurately described something real that no longer exists, because the retrieval step handed it stale information and the prompt contained no instruction to prefer recent sources or flag version uncertainty.
This failure was not invisible. A PM reading traces as a regular activity would have seen, across multiple sessions, a pattern of retrieved chunks with old timestamps being used to answer questions about current APIs. That pattern is a product signal. It is a signal that the retrieval configuration, the recency weighting, the index freshness cadence, the version-filtering logic, was not specified as a product requirement.
The model did exactly what it was set up to do. The setup was not specified well enough to prevent this class of failure.
That is not an engineering failure. That is a spec gap.
The Spec Gap as Root Cause
The uncomfortable position this article holds: most AI failures that get attributed to the model are actually failures of specification that happened earlier, in rooms where the PM was present.
A spec gap in an AI feature is not the same as a missing requirement in a traditional feature. In traditional software, an unspecified behavior usually produces an error or a null state, something obviously broken. In an LLM-powered feature, an unspecified behavior produces a confident, fluent, wrong answer. The model fills the gap. It fills it with whatever its training and retrieved context make most plausible.
That means the gap is invisible until the output is wrong.
The three most common spec gaps that trace analysis reveals:
Unspecified scope boundaries. The prompt tells the model what it can do. It does not tell the model what it cannot do. When a user asks a question outside the intended scope, the model answers anyway. The trace shows a system prompt with no constraint on domain, and a user query that wandered outside it.
No context injection rules. The product retrieves external documents and injects them into the context window. The spec never defined what makes a retrieved document acceptable, how recent it must be, what sources are authoritative, what happens when no relevant document is found. The trace shows the model reasoning from whatever happened to rank highest in the retrieval step.
Implicit output format expectations. The product team expected structured output. The prompt did not require it. The model sometimes produces structure, sometimes prose. The trace shows the same prompt producing inconsistent shapes, not because the model is unreliable, but because it was never constrained.
None of these gaps are fixed by retraining the model. They are fixed by writing a better spec and encoding that spec into the prompt and retrieval configuration.
The Judgment Turn
Here is what trace analysis actually reveals about the PM role in AI product development.
Most of the failure surface is not in the model. It is in the instructions given to the model, the context provided to the model, and the absence of constraints on what the model is allowed to do. All three of those are product decisions. They are made by the person writing the spec, designing the retrieval pipeline, or approving the system prompt.
When a PM files a ticket that says "model hallucinated" and moves on, they are declining to own the part of the failure that is actually theirs. That is not negligence, it is a gap in how AI PM work is currently understood. The model is a black box with a reputation for strange behavior, and handing the ticket to engineering feels like the correct escalation path.
Trace analysis breaks that reflex. When you read a trace, the failure becomes concrete and locatable. It stops being "the AI did something weird" and becomes "the system prompt contained no instruction about version constraints and the retrieved context was indexed eight months ago." That is a sentence a PM can act on.
Reading traces is not a deep technical skill. It requires comfort with structured logs, an understanding of what each layer of the system is responsible for, and the discipline to do it before the post-mortem, not just during it. Teams that make trace review a regular PM activity, not a post-incident activity, catch spec gaps before they accumulate into user-facing failures.
The question worth sitting with: if you shipped an AI feature in the last six months, when did you last read a trace from it?
Key Takeaways
- A trace is a deterministic record of every layer between the user's input and the model's output, the PM's primary reading zone is the prompt and the context window contents, not the token metadata.
- "The model hallucinated" describes the output, not the cause, trace analysis almost always locates the cause in a prompt that lacked constraints or a retrieval step that injected stale or irrelevant context.
- The failure pattern in code assistant systems, suggesting deprecated methods, is directly traceable to retrieval index freshness and missing version-scoping instructions, both of which are product specification responsibilities.
- The three spec gaps that traces reveal most often are: unspecified scope boundaries, missing context injection rules, and implicit output format expectations.
- Trace review is a preventive PM activity, not only a post-incident one, teams that read traces regularly catch failure classes before they reach users.