A system that is right 80% of the time can be wrong where it matters most.
The AI Product Requirements Document, Writing Specs for Systems That Might Be Wrong
How to specify AI-powered features when the output is probabilistic, not deterministic, defining acceptable variance, safety bounds, and failure modes instead of expected behavior. A Product Requirements Document without a section on acceptable error rate is not a Product Requirements Document, it is a demo script.
One-line definition: A specification format for AI-powered features that replaces expected outputs with acceptable outcome ranges, variance budgets, and tail-failure definitions, because a system that is right 80% of the time can still be wrong in the 20% that matters most.
The Demo Works. The Product Does Not.
Here is what actually happens on most teams. A product manager writes a spec. The spec describes the feature working. QA tests the feature working. The demo is the feature working. The team ships.
Then real users arrive with real documents, real edge cases, and real expectations. The model produces outputs the team never considered. The product manager calls it a bug. Engineering calls it model behavior. The conversation goes in circles because the specification never defined what "wrong" looks like.
This is not an engineering failure. It is a specification failure. The document described a happy path at median performance and called it a product requirement.
Why Traditional Acceptance Criteria Breaks Down
A traditional acceptance criterion reads like a contract. "When the user submits a support ticket, the system returns a suggested response within three seconds." Either it does, or it does not. The criterion has one valid state: pass.
An AI feature does not work this way. "When the user submits a support ticket, the system returns a suggested response", and that response might be excellent, adequate, misleading, or confidently wrong. The system returns something within three seconds every time. The acceptance criterion passes every time. The product fails in ways the criterion cannot see.
The underlying problem is that acceptance criteria assume a deterministic function. Given input X, output Y. AI systems are probabilistic functions. Given input X, the output is a distribution. A specification that ignores the distribution is not specifying the product, it is specifying the best case.
What Replaces Acceptance Criteria: Outcome Envelopes and Variance Budgets
An outcome envelope defines the full range of acceptable outputs across the probability distribution, not just the expected output at the median. It has three parts.
The first part is the target band: what the output should look like at the 50th percentile. This is the closest thing to a traditional acceptance criterion. It describes the feature working as intended.
The second part is the acceptable variance boundary: how far from the target the output can drift before it constitutes a product failure. This is expressed as a threshold at a specific percentile. "At the 95th percentile, the summary must still contain the primary decision from the source document." This is not a median requirement. It is a tail requirement.
The third part is the hard floor: the output condition that is never acceptable, regardless of percentile. "The system must never present a summary that contradicts a factual claim in the source document." This is a safety bound, not a performance target. It does not have a percentile. It is binary.
A variance budget translates the outcome envelope into a number the team can measure. It answers: how much degradation at the tails is acceptable before the feature should not ship? "False negative rate below 8% at the 95th percentile on the held-out evaluation set" is a variance budget. "Works well most of the time" is not.
The Comparison: Deterministic Spec vs. Probabilistic Spec
| Dimension | Deterministic Spec | Probabilistic Spec |
|---|---|---|
| Success definition | Output matches expected value | Output falls within the defined outcome envelope at target percentiles |
| Failure definition | Output does not match expected value | Output breaches the variance budget or crosses a hard floor |
| Edge case handling | List of specific inputs with expected outputs | Distribution of input types with acceptable output ranges per type |
| Quality Assurance method | Pass/fail test against expected outputs | Evaluation on held-out set measuring distribution of outcomes across percentile bands |
| Launch criteria | All acceptance criteria pass | Variance budget thresholds met; hard floors hold; human review rate within acceptable range |
| Post-launch monitoring | Error rate (binary: did it work?) | Drift monitoring across percentile bands; tail failure rate tracked separately from median performance |
The column that matters most is the launch criteria row. A deterministic system ships when all acceptance criteria pass. A probabilistic system ships when the distribution of outputs falls within the defined envelope, and that envelope must be defined before the first model evaluation, not after.
The Notion Case: A Product Decision Disguised as a Model Limitation
Notion AI added document summarization as one of its early features. At the median, a reasonable business document, meeting notes, a project update, the summaries work. They capture the main point. They are useful.
At the tails, the behavior changes in ways that matter specifically for the users who need AI help most. Feed Notion AI a technical specification with conditional logic ("if the user has not completed onboarding, the feature is disabled except in these three cases") and the summary drops the conditionals. It presents the general rule without the exceptions. A user who trusts that summary makes decisions based on an incomplete model of the system.
Notion's response was not to define an acceptable error rate for conditional logic preservation. It was not to add a confidence score, or to restrict summarization on documents above a certain technical complexity, or to surface a different User Interface treatment when the document contained conditional structures. The response was microcopy. "Review carefully." Two words placed near the output.
That is a product decision. It is a reasonable product decision, Notion is not a document verification tool, and the feature is positioned as a time-saver, not a source of truth. But it is a decision. A product manager made a call about the acceptable tail behavior and encoded it as a User Interface hedge rather than a product constraint.
The problem is that the decision appears nowhere in the specification. There is no written statement of what the acceptable error rate is for conditional logic. There is no percentile threshold for summary completeness. There is no definition of what "review carefully" is meant to catch. The microcopy is a symptom of an unspecified failure mode, a product team that discovered the p95 behavior after shipping and chose a response without a documented rationale.
The Failure Modes Section: What to Specify Before the Model Is Trained
Every AI Product Requirements Document needs a failure modes section. It belongs before the model is built, not after the model is evaluated. Writing it after evaluation is not specification, it is post-hoc rationalization.
The failure modes section answers four questions.
What does failure look like at the median? Not a bug in the traditional sense, a systematic pattern of inadequate outputs at normal operating conditions. For a summarization feature: "Summary omits the primary action item in documents where the action item appears in the second half of the document."
What does failure look like at the tails? The output condition that appears rarely but causes disproportionate damage. For a summarization feature: "Summary contradicts a factual claim in the source document, causing the user to act on incorrect information." This is the hard floor condition.
Who is harmed by each failure mode, and how? This is not a philosophical question, it is a product scoping question. A user who receives an incomplete summary is mildly inconvenienced. A user who receives an internally contradictory summary and acts on it before reviewing the source document may make a wrong decision. The severity of harm determines the variance budget. Higher severity means tighter tolerance.
What does the system do when it detects it might be in a failure mode? This is the hardest question and the one most specifications skip entirely. The answer is not "surface a disclaimer." The answer requires the team to decide: does the system surface a lower-confidence output with a signal, withhold the output and ask for human review, or present the output without any signal and accept the tail risk? Each choice has a product implication. Each choice should be in the specification.
The Judgment Turn
Most teams write AI Product Requirements Documents that describe the happy path at p50, then ship, discover the p95 behavior, and call it a bug. It is not a bug. It is an unspecified requirement.
The uncomfortable position is this: a Product Requirements Document that does not define acceptable error rate, tail failure conditions, and hard floors is not a Product Requirements Document for an AI feature. It is a Product Requirements Document for a demo. The demo is always at p50. Production is not.
The instinct to avoid specifying failure modes is understandable. Specifying them requires the team to make explicit decisions about acceptable harm. That is uncomfortable. It is easier to ship and respond. But shipping without a specified variance budget means the team cannot distinguish between a model underperforming and a product performing exactly as it was implicitly designed to, with no floor.
Notion's "review carefully" microcopy is not wrong. It is honest about the product's limits. What is missing is the document where someone wrote down: "We accept that the model will drop conditionals at the tail. Here is the rate we find acceptable. Here is why. Here is what we will monitor." That document does not exist, which means the decision cannot be revisited with data. The next product manager who works on that feature inherits a product decision encoded only in two words of microcopy.
Write the failure modes section before the model is trained. Define the outcome envelope before the first evaluation. Set the variance budget before you see the distribution. If you wait until you see the distribution, you will rationalize whatever you see.
Key Takeaways
- Acceptance criteria assume deterministic outputs. AI features produce distributions. A specification that does not address the distribution is not specifying the product.
- An outcome envelope has three parts: the target band at the median, the acceptable variance boundary at a specific percentile, and the hard floor, the condition that is never acceptable.
- A variance budget is the launch criterion for a probabilistic system. "False negative rate below 8% at the 95th percentile on the held-out evaluation set" is a variance budget.
- The failure modes section must be written before the model is trained, not after evaluation. Post-hoc failure documentation is not specification, it is rationalization.
- When a team ships without a defined hard floor, it cannot distinguish between a model underperforming and a product performing exactly as implicitly designed, with no defined lower bound.