The evals passed. The product failed. These are not the same thing.
AI Evals Are a Product Decision, Not an Engineering One
How to design quality rubrics and golden datasets that actually measure whether your AI feature does the right thing, not just the technically correct thing. If you do not know what is in your golden dataset, you do not know what you are shipping.
Definition: Evaluation design for AI features is the process of deciding what "good" means for your model's outputs, and building the datasets and rubrics to prove it.
The Setup Most Teams Get Wrong
You have a new AI feature in review. Engineering runs evals. The numbers look good, 94% accuracy on the test set, average latency under two seconds, no schema violations. The feature ships.
Three weeks later, your support queue fills with users reporting that the AI gave them confident, fluent, completely wrong answers.
The evals passed. The product failed. These are not the same thing.
This gap is not an engineering failure. It is a product failure, specifically, a failure to design evals around what users actually need rather than what the system can be easily graded on.
Two Kinds of Evals, One Dangerous Conflation
There are two structurally different questions you can ask about an AI output.
The first question: Did the output match the expected format, schema, or reference answer? This is what automated evals answer. They are fast, cheap, and scalable. They are also blind to anything that requires reading the output as a human would.
The second question: Did the output actually serve the user in this context? This is what quality evals answer. They require judgment, rubrics, and, at some stage, humans reading real outputs in real situations.
Most teams run the first kind of eval because it is tractable. They call it "eval coverage." It is not. It is format coverage. These two things are not synonymous, and treating them as such is how you ship AI features that are technically correct and experientially broken.
What Each Eval Type Catches, and What It Misses
| Dimension | Automated Evals | Human-in-the-Loop Evals |
|---|---|---|
| What it checks | Schema, format, citation presence, latency, token count | Response quality, user appropriateness, factual accuracy in context, tone |
| Speed | Seconds per run | Hours to days per batch |
| Cost | Near-zero at scale | High, requires trained reviewers or PM time |
| When to use | Every pull request, every model update, continuous regression | Pre-launch quality gate, post-launch sampling, novel failure investigation |
| What it misses | Confident-but-wrong outputs, harmful but grammatically correct responses, contextually inappropriate answers | Volume; cannot sample every response in production |
| Failure mode it creates | False confidence that the system is working | Slow feedback loops; reviewer fatigue skews toward surface-level issues |
The table above is not a recommendation to choose one. You need both, and you need to understand which failure modes each one is blind to. The dangerous move is running only automated evals and calling that "tested."
The Pattern Intercom's Fin AI Agent Illustrates
Consider how a customer support AI like Intercom's Fin agent faces this problem. The pattern is instructive even if the internal implementation is not publicly documented in detail.
A support AI that answers customer questions has two layers of evaluation pressure working against each other. The first layer is structural: did the response cite a knowledge base article? Did it stay within the allowed response length? Did it avoid prohibited phrases? These are automatable. You write a checker, run it on every response, get a pass or fail.
The second layer is the one that determines whether customers actually trust the product: was the answer right? Was it confident when it should have been uncertain? Did it escalate when it did not know, or did it fabricate a plausible-sounding resolution that sent the customer in circles?
A pattern consistent with mature AI product teams is layered evals: automated checks handle the first layer at scale, and a human review queue specifically targets the "confident but wrong" failure mode that automated evals will never catch. The human queue is not random sampling, it is deliberately seeded with cases where the model expressed high confidence. That is the category of outputs most likely to damage user trust, and it is the category most automated systems are structurally blind to.
The judgment call baked into that design, that high-confidence errors are more damaging than hedged errors, is a product decision. It required someone to decide what failure mode was most costly to users. That decision shapes the entire evaluation architecture.
The Golden Dataset Is a Product Artifact
Every evaluation system runs against a dataset of known inputs and expected outputs. This is your golden dataset. It is the ground truth your evals are graded against.
Here is what most PMs do not realize: the golden dataset is not a technical artifact. It is a product artifact. It encodes your acceptance criteria for what "good" looks like. Every input you include or exclude shapes what your evals are sensitive to. Every expected output you label represents a judgment call about what the right behavior is.
If engineering builds the golden dataset without PM involvement, they will populate it with cases that are easy to label and easy to grade. Edge cases that require domain knowledge or contextual judgment will be underrepresented, not because engineers are cutting corners, but because those cases are genuinely hard to label without product context.
The result is an eval suite that grades the system on the easy cases and ignores the hard ones. Your numbers will look strong. Your product will fail at the exact moments users need it most.
The Uncomfortable Position
Delegating eval design to engineering is the same as delegating acceptance criteria. It will optimize for what is easy to measure, not what matters to users.
This is not a critique of engineering. Engineers are excellent at building rigorous, scalable evaluation pipelines. They are not well-positioned to decide what the system should be doing when a user's question falls outside the training distribution, when tone matters more than factual accuracy, or when the right answer is to say "I do not know" rather than generate a fluent response.
Those decisions require product judgment. They require someone who has talked to users, who understands the failure cost asymmetry, who knows which errors users will forgive and which ones will end the relationship.
If you are not in the room when the golden dataset is designed, someone else is making those calls on your behalf. They are making them based on what they can automate, not what you have learned about your users.
How to Take Ownership Without Becoming an ML Engineer
You do not need to write evaluation code to own eval design. You need to do three things.
Define the failure hierarchy before any dataset is built. For your specific AI feature, rank the failure modes. A wrong answer delivered with high confidence is almost always worse than no answer. An answer that escalates when it should not is almost always better than an answer that resolves when it cannot. This ranking is your input to the eval design, and it changes what the golden dataset needs to cover.
Write the rubric before reviewing the outputs. A rubric is not a checklist. It is a set of criteria with explicit tradeoffs. For a support AI, a rubric might say: accuracy on disputed facts matters more than completeness; completeness matters more than brevity; brevity matters more than formality. That ordering guides human reviewers and prevents drift across batches. It also makes your quality standard legible to engineering so they can build automated proxies for each criterion.
Audit the golden dataset at launch and at every major model update. Pull a sample, twenty to fifty examples. Read the inputs. Read the expected outputs. Ask whether the distribution of cases represents what your actual users encounter, not what was convenient to label. If the hard cases are absent, you do not have a golden dataset. You have a collection of easy wins.
The Judgment Turn
Everyone says evals are important. Most teams run format checks and call it done.
The gap between those two positions is not laziness. It is a structural misunderstanding of who evals are for. Automated evals are for the engineering team, they tell you whether the system is behaving predictably. Quality evals are for the user, they tell you whether the system is doing something worth shipping.
You cannot get the second kind of signal from the first kind of evaluation. And you cannot design the second kind of evaluation without knowing what your users consider a failure.
If you do not know what is in your golden dataset, you do not know what you are shipping. You are relying on someone else's definition of good, someone who did not do your user research, does not carry your escalation path, and will not be answering support tickets when the confident-but-wrong answers go out.
That is not an engineering problem. It is yours.
Key Takeaways
- Automated evals check format and schema compliance. They do not measure whether the output served the user, that requires a separate quality eval layer with human review.
- The golden dataset is the living definition of your AI feature's acceptance criteria. If you are not involved in building it, you have outsourced your acceptance criteria.
- High-confidence wrong answers are the failure mode most automated systems miss and most damaging to user trust. Designing specifically for this failure mode is a product decision.
- Your rubric, the explicit ranking of failure modes, should exist before any eval dataset is built. It guides both human reviewers and automated proxy design.
- Audit your golden dataset at launch and at every major model update. Distribution gaps are where your evals lie to you.