If your AI feature shipped without these elements, it was released, not launched.
Rewriting Done, Completion Criteria When the Output Is Probabilistic
Traditional Definition of Done breaks for AI features because quality assurance can verify the feature ran, not that it ran well. This article shows how to define completion criteria when success is a distribution of outputs, not a pass/fail test.
How to redefine agile completion criteria for AI features where success is a distribution of outputs, not a pass/fail test.
The Scene That Breaks the Old Playbook
Your team builds an AI-powered feature. Engineering confirms it runs without errors. Quality assurance runs test cases and marks them passed. The feature ships. Two weeks later, support tickets accumulate around edge cases your test suite never covered because your test suite verified execution, not quality.
This is not a hypothetical. It is the default outcome for teams that apply a deterministic Definition of Done to a probabilistic system. The old playbook has a pass/fail assumption baked into every checkpoint. An AI feature does not pass or fail. It produces a distribution, and where you draw the line on that distribution is a judgment call, not a configuration setting.
The question is not whether your AI feature works. The question is: what percentage of the time does it work well enough, and who decided what "well enough" means?
flowchart TD
A[What type of output?] --> B[Deterministic]
A --> C[Probabilistic AI]
B --> D[Pass or Fail]
D --> E[Binary Done]
C --> F[Quality Floor]
C --> G[Sampling Protocol]
C --> H[Launch Threshold]
C --> I[Rollback Trigger]
F & G & H & I --> J[Probabilistic Done]Why Traditional Definition of Done Fails Here
A conventional Definition of Done is built for systems where correct behavior is enumerable. The login form either accepts valid credentials or it does not. The payment API either returns a confirmation or it returns an error. You can write a test. You can run the test. You can read the result.
A language model does not work this way. Given the same prompt, it will produce different outputs across runs, across model versions, and across input variations you did not anticipate when you wrote the test. Quality assurance can verify that the model responded. It cannot verify that the response was accurate, appropriate, or within the quality band your users expect, unless you define that band explicitly and build a protocol to measure against it.
Most teams do not define that band. They define that the feature exists.
The result is a Definition of Done that confirms shipping without confirming readiness. You can mark the ticket done, close the sprint, and deploy a feature that produces excellent outputs thirty percent of the time and mediocre outputs seventy percent of the time, and your completion criteria will never flag this. Every checkbox will be green.
The Google Bard Demo, A Launch With a Date, Not a Standard
In February 2023, Google publicly launched Bard, its conversational AI product, under direct competitive pressure from the release of ChatGPT and the Microsoft-OpenAI partnership announcement. The first public promotional material included a factual error: Bard incorrectly stated that the James Webb Space Telescope took the first pictures of an exoplanet outside our solar system. Astronomers and journalists identified the error within hours. The error appeared in a promotional asset that Google itself produced and distributed.
Google's own Chief Executive Officer, Sundar Pichai, acknowledged the mistake publicly. The market response was swift. Alphabet's share price dropped significantly in the days following the demo, with multiple financial outlets citing the error as the proximate cause. The exact figures varied by reporting date and source, so this article does not pin a specific percentage, but the directional response was unambiguous and extensively documented.
The Bard launch is not a story about a careless team. Google's AI research organization is among the most sophisticated in the world. It is a story about what happens when a launch date becomes the primary completion criterion. The competitive pressure to ship before a rival was real. The internal calculus that led to proceeding without catching the error in the promotional asset was, at minimum, a failure of the quality gate, not the engineering infrastructure.
This is the risk you accept when your Definition of Done is a date. A date is always achievable. A quality standard is conditional.
Deterministic Done vs. Probabilistic Done
The table below is not a theoretical construct. It reflects the actual difference between what most teams do and what an AI feature requires.
| Dimension | Deterministic Definition of Done | Probabilistic Definition of Done |
|---|---|---|
| Quality threshold | Binary, the feature works or it does not | Explicit floor, at least X% of outputs must score above Y on a defined rubric |
| Sample size for sign-off | Test suite cases (often tens to hundreds) | Statistically meaningful sample drawn from representative input distribution (often hundreds to thousands) |
| Who owns quality judgment | Engineering and quality assurance | Product manager owns threshold definition; human raters or automated evaluation owns measurement |
| How failure is defined | Feature does not execute, or throws an error | Output quality falls below threshold in production, even if the feature executes without errors |
| Rollback trigger | System error rate exceeds tolerance | Production quality metric drops below floor, defined before launch, not after problems appear |
| Sprint review evidence | Demo of the feature working | Distribution summary, median output quality, variance, and worst-case percentile shown explicitly |
| Post-launch monitoring | Uptime and error rate | Quality metric tracked over time, with owner and escalation path |
Every column in the right side requires a decision that most product managers have not been asked to make explicitly. The probabilistic Definition of Done does not emerge on its own. Someone has to define the threshold, design the sampling protocol, assign ownership of the quality judgment, and name the rollback trigger before the feature ships. That someone is the product manager.
The Sprint Review Problem
Here is the uncomfortable position this article is willing to hold: you cannot demo an AI feature at a sprint review and call it done if you do not know what percentile of outputs you are showing.
A sprint review demo is a single output from a distribution. If the product manager or engineer selects an output that looks good for the demo, they are showing the stakeholder a cherry-picked sample. Stakeholder approval is then approval of a best case, not a representative product. This is not deception in the conventional sense, it is the natural human tendency to present work favorably. But for AI features, it produces a structural information gap at the exact moment when go/no-go decisions are made.
The alternative is not to show a bad output in the demo. The alternative is to present the distribution honestly.
A well-run sprint review for an AI feature shows three things: the best-case output, the median output, and the worst-case output from a defined sample. It names the percentage of outputs that fall above the quality floor. It states whether that percentage meets the launch threshold or not. The stakeholder then makes a go/no-go decision with visibility into the full shape of the product, not just its highlight reel.
This is harder. It exposes gap between "the demo looks impressive" and "the product is ready." It often surfaces conversations that teams would prefer to defer. That is precisely why it is the right practice.
Building the Probabilistic Definition of Done
The mechanics are less complex than they appear. Four elements are required.
A quality floor with a definition. Not "outputs should be good." A specific criterion: factually accurate, does not hallucinate product names, matches the reading level of the target audience, or passes a structured rubric scored by human raters. The definition must be specific enough that two independent raters would reach the same score on the same output at least eighty percent of the time.
A sampling protocol. The sample must cover the distribution of real inputs, not the inputs the team finds convenient to test. This means drawing from actual or simulated user queries, including edge cases, ambiguous inputs, and adversarial inputs. The sample size must be large enough to produce a meaningful estimate of the distribution, typically no fewer than two hundred outputs for an initial sign-off, with ongoing monitoring in production.
A launch threshold. A specific percentage of outputs that must meet the quality floor before the feature ships. This is a judgment call, and it should reflect the risk tolerance of the product context. A consumer-facing health information feature demands a different threshold than an internal productivity tool. The threshold is set before evaluation begins, not adjusted after seeing the results.
A rollback trigger. A production metric and a floor, defined in advance. If the quality metric in production falls below the rollback floor, the feature is pulled back or disabled until the problem is diagnosed. This trigger is not optional for a feature that was released under competitive pressure without a full sign-off. It is the safety net that a missing quality gate requires.
The Judgment Turn
If your AI feature shipped without these four elements, it was released, not launched. A launch has success criteria. A release has a date.
This distinction is not semantic. It determines whether you have accountability for what the feature does after it ships. A release transfers ownership to the user. A launch retains it with the team, because the team defined what acceptable looks like and committed to monitoring against it.
The discomfort here is real. Defining a quality floor and a launch threshold often means surfacing the probability that the current model does not meet it. That conversation is easier to avoid. You can ship, watch, and respond when problems emerge. Many teams do exactly this and call it iteration.
The cost is that users experience the distribution before you have characterized it. The rollback happens in response to visible damage rather than in anticipation of it. The sprint review showed a great demo. The product, in the hands of real users, showed something else.
You own both.
Key Takeaways
- Traditional Definition of Done verifies execution. It cannot verify quality when the output is a distribution. These are different problems requiring different tools.
- The Google Bard launch in February 2023 is a documented case of a launch date functioning as the completion criterion, with a publicly visible quality failure in the first promotional material.
- A probabilistic Definition of Done requires four explicit decisions before the feature ships: a quality floor with a definition, a sampling protocol, a launch threshold, and a rollback trigger.
- A sprint review demo for an AI feature is a single output from a distribution. Showing only the best-case output is not evidence of launch readiness. Showing the distribution, best case, median, worst case, and the percentage above the quality floor, is.
- If you shipped without a quality floor and a rollback trigger, define them now for the live feature. Post-launch quality gates are weaker than pre-launch gates, but they are better than none.