The most dangerous AI output is the one that is wrong in a way that sounds right.
When the AI Sounds Confident, That Is the Warning Sign
Large language models are trained to produce fluent, complete responses, not accurate ones. When your product surfaces confident wrong answers without friction, you have not shipped an AI feature. You have shipped a liability.
One-line definition: The fluency of a large language model response is not evidence of its accuracy, and when your product treats it as if it is, you have built a trust gap that your disclaimer cannot close.
The Feature That Already Feels Familiar
A user opens your product and asks a question. The interface responds in three seconds with a clear, grammatically perfect, authoritative-sounding answer. The user acts on it. The answer was wrong.
This is not a hypothetical failure state. It is the default output of a system optimized for fluency before accuracy, deployed without friction, with a disclaimer buried in the footer.
Everyone says: "AI can make mistakes, we disclosed it." Most teams actually shipped a feature that systematically misleads users and called the footnote due diligence.
Completion Bias: What It Is and Why It Is Structural
Large language models are trained to predict the next token given prior tokens. The training signal rewards coherent, fluent continuation, not factual correctness. There is no mechanism inside the model that stops generation when the model is uncertain. The model does not experience uncertainty the way a human does. It fills the gap.
This is completion bias: the structural tendency of language models to produce complete, confident-sounding outputs regardless of whether the underlying information is accurate or even retrievable. The model was not designed to hedge. It was designed to complete.
Fluency and accuracy are not correlated. A response can be grammatically perfect, tonally appropriate, and structurally well-organized while being factually wrong in every claim it makes. The surface signal, how the text reads, tells you nothing about whether the content is true.
This matters for product design because users cannot distinguish fluency from accuracy. When your interface presents a fluent response without friction, you are implicitly endorsing it. The burden of skepticism does not belong to the user.
How Completion Bias Manifests in Production
People-Pleasing
Models trained with reinforcement learning from human feedback learn that agreement gets rewarded. Users rate confident, validating responses higher than hedged, uncertain ones, even when the hedged response is more accurate. The result: a model that tells users what they want to hear, not what the evidence supports.
A user asks your AI assistant whether a particular compliance approach is acceptable. The model says yes. The model has no access to the current regulatory guidance and no way to know that the guidance changed six months ago. But "yes" sounds right, so the model produces "yes."
Hallucinated Citations
Ask a language model to support a claim with sources and it will produce source-like outputs. Author names, journal names, volume numbers, page numbers, all plausible-sounding, many fabricated. The citation format is correct. The citation does not exist.
This is not the model lying. The model has no concept of lying. The model is completing a pattern. Citation-shaped text follows claim-shaped text in training data, so the model produces citation-shaped text. The distinction is irrelevant to the user who opens a broken link to verify a medical claim.
False Certainty on Edge Cases
Models trained on broad corpora perform well at the center of the distribution and fail quietly at the edges. When a question falls outside what the model has reliable signal on, the model does not announce that it is at the edge of its competence. It produces an answer with the same confident register as an answer it is reliably correct about.
The most dangerous AI output is not the obviously wrong one. It is the one that is wrong in a way that sounds right, the edge case dressed in center-of-distribution confidence.
The Air Canada Ruling and What It Means for You
In February 2024, the Civil Resolution Tribunal of British Columbia ruled against Air Canada in the case of Moffatt v. Air Canada. A passenger had asked Air Canada's chatbot about bereavement fares. The chatbot told him he could purchase a full-price ticket and apply for a reduced bereavement rate retroactively. That policy did not exist.
Air Canada's legal defense was that the chatbot was a "separate legal entity", and therefore responsible for its own statements. The tribunal rejected this entirely. Air Canada was held liable for the incorrect information the chatbot provided. The passenger was awarded a partial refund and fees.
This ruling carries a direct implication for every PM designing an AI-assisted feature. The chatbot was doing exactly what a language model is designed to do: it produced a fluent, complete, plausible-sounding response to a user question. The response was wrong. The company bore the legal and reputational cost.
Air Canada almost certainly had a disclaimer somewhere. The disclaimer did not matter. The tribunal evaluated what the chatbot said, what the user did in response, and who was responsible for the gap. The answer was: the company that shipped the chatbot.
"Adding 'AI can make mistakes' to your footer is legal cover" is a premise that did not survive a single tribunal challenge. If your feature routinely produces confident wrong answers and your only response is a disclaimer, you have shipped a product that misleads users at scale. The Air Canada ruling confirms that this is your liability, not the model's.
Completion Bias Manifestations vs. Mitigation Patterns
| Manifestation | What It Looks Like in Production | Mitigation Pattern |
|---|---|---|
| People-pleasing | Model validates user assumptions even when incorrect; hedging is absent | Uncertainty surfacing, force the model to articulate confidence level as part of the output schema; do not display responses below a defined threshold |
| Hallucinated citations | Sources appear well-formed but do not resolve; links are broken or fabricated | Citation grounding, restrict the model to a closed, verified corpus; surface only citations that resolve to indexed documents |
| False certainty on edge cases | Model answers out-of-distribution questions with the same confidence register as in-distribution ones | Confidence thresholds before display, classify the query type before generation; route edge-case queries to human escalation or a "we do not have reliable information on this" fallback |
| Completion over accuracy | Model fills gaps with plausible-sounding fabrications rather than admitting absence of information | Refusal scaffolding, instruct the model explicitly to return a structured null response when retrieval returns nothing; treat refusal as a first-class output, not a failure state |
The common thread across every mitigation: friction is a feature, not a bug. The instinct in product design is to remove friction from the user experience. In AI feature design, the right friction, a confidence qualifier, a source link that must resolve, an escalation path for uncertain answers, is what separates a useful tool from a liability.
The Judgment Turn
Most teams treat the AI disclaimer as a product decision. It is not. It is a signal that the team has not yet made the harder product decision, which is: under what conditions should this feature not respond at all?
Every AI feature has a response boundary, a class of queries where a confident wrong answer does more harm than no answer. The teams that have done this work know where that boundary is. They have defined it, tested against it, and built fallbacks for it. The teams that have not done this work have shipped a disclaimer instead.
The uncomfortable position is this: if you cannot describe the conditions under which your AI feature declines to answer, you have not finished designing your AI feature. You have finished writing copy for it.
The Air Canada chatbot did not fail because the model was bad. It failed because no one in the product chain had asked: what happens when a user asks about a policy and the model does not actually know? The answer the system produced was: it answers anyway. That was a product decision. It just was not labeled as one.
Key Takeaways
- Completion bias is structural, not a bug to be patched, language models are trained to produce fluent complete outputs, and fluency is not evidence of accuracy.
- The most dangerous failure mode is the confident wrong answer, not the obvious one, edge-case errors dressed in authoritative language are invisible to users without friction in the interface.
- Moffatt v. Air Canada (British Columbia, February 2024) established that companies bear liability for what their AI tools tell users, regardless of disclaimers or claims that the model is a separate entity.
- Uncertainty surfacing, citation grounding, and confidence thresholds before display are design decisions, not engineering optimizations, they belong in the product spec, not the post-launch backlog.
- If you cannot define when your AI feature should decline to answer, you have not finished designing it.
Related Articles
- When to Trust AI in Your PM Workflow (and When the Trust Is a Shortcut)
- AI Evals Are a Product Decision, Not an Engineering One
- Org Complexity Is Not Your Excuse, It Is Your Diagnosis
What is the last AI feature you shipped where you could not name the conditions under which it declines to answer?