Legal said no to the version we built without asking them.
AI in Regulated Industries, The Compliance Layer Is a Product Decision
Most AI features killed by legal were not killed by compliance, they were killed by product teams that treated compliance as a review step instead of a design input. Here is how to stop building features that die in legal review.
One-line definition: Shipping AI in healthcare, finance, or insurance is not a legal obstacle course, it is a product design problem where regulatory constraints are inputs, not exits.
The Room Where Features Die
A team at a mid-sized fintech spends three months building an AI-powered loan recommendation feature. The model is well-calibrated. The user research is solid. The engineering is clean. Six weeks before launch, legal reviews it and kills it.
Not because AI is banned. Not because the feature is illegal. Because the model cannot explain why it made a recommendation in a way that satisfies the fair lending audit requirement. The team did not know that requirement existed. Legal did not know the feature existed until the review meeting.
Three months. One meeting. One veto.
That is not a compliance problem. That is a discovery problem dressed up as a compliance problem.
Three Failure Modes, and What Actually Causes Each
Every team shipping AI in regulated industries fails in one of three ways. The root cause is almost never "regulations are too strict." It is almost always a process failure that regulation exposed.
Failure Mode One: Shipping Nothing
The team hears "AI in healthcare" and immediately imagines HIPAA violations. Someone mentions GDPR. Legal gets involved early but without a specific feature in scope, so the conversation is theoretical. Theoretical legal conversations produce theoretical vetoes. The feature never ships.
The root cause here is not excessive caution, it is asking legal the wrong question. "Can we use AI?" produces a different answer than "Can we use AI to surface historical claim data to a logged-in user who has already consented to data processing, with no third-party data sharing?" The first question invites a veto. The second invites a conversation.
Failure Mode Two: Compliance Surprise
The team builds first and asks later. The feature is real, the investment is sunk, and legal sees it for the first time at the review gate. Whatever legal finds wrong now requires either a rebuild or a kill, because the feature was not designed with the constraint in mind, the constraint cannot be bolt-on fixed without unwinding the architecture.
This is the most expensive failure mode. It is also the most preventable. The feature was not killed by compliance. It was killed by the sequence: build, then ask.
Failure Mode Three: Compliance Theater
This is the hardest failure mode to see clearly, because it looks like a win. The feature ships. Legal is satisfied. The team celebrates.
But the feature that shipped is not the feature that was designed. The model was stripped of the training data that made it accurate. The recommendation engine was replaced with a rules-based fallback that legal could audit. The explainability requirement was solved by showing users a generic disclaimer instead of actual reasoning.
The user problem is unsolved. The compliance box is checked. Nobody in legal did anything wrong, they reviewed what they were given. The product team accepted terms that made the feature useless rather than fighting for a version that was both compliant and valuable.
Compliance as Blocker vs. Compliance as Design Constraint
The difference between teams that ship AI in regulated industries and teams that do not is not risk tolerance. It is when and how they involve compliance.
| Dimension | Compliance as Blocker | Compliance as Design Constraint |
|---|---|---|
| When legal enters | At review, after build | At discovery, before design |
| What legal is asked | "Is this okay?" | "What does this need to be true for this to be okay?" |
| Feature capability | Full capability built, then stripped | Capability designed around the constraint from the start |
| Time to ship | Longer (rebuild cycles after veto) | Shorter (no late-stage rework) |
| User experience | Either nothing or a neutered version | A constrained version that still solves the core problem |
| Regulatory outcome | Unpredictable | Known before engineering starts |
| Legal relationship | Adversarial (legal as gatekeeper) | Collaborative (legal as design partner) |
The teams that treat compliance as a design constraint do not ship faster because they have better lawyers. They ship faster because they have fewer rebuild cycles. The constraint is baked into the product brief. Engineering builds to the constraint once instead of building without it and then ripping out what does not pass.
The JPMorgan COiN Case, Explainability as Product Design
In 2017, Bloomberg reported that JPMorgan Chase's Contract Intelligence system, referred to as COiN, reviewed commercial loan agreements in seconds, handling work that had previously consumed 360,000 hours of lawyer and loan officer time annually across the bank. The coverage focused on efficiency. The more instructive part of the story received less attention.
According to that reporting, one of the design requirements for COiN was that the system needed to produce outputs that were auditable. Regulatory compliance in commercial lending requires that decisions, or inputs to decisions, can be explained and reviewed. A black-box model that reduced contract review time but could not show its work would not have cleared the regulatory bar required for deployment at that scale.
Whether the explainability architecture was driven by product, legal, or their intersection is not publicly confirmed. What the public record suggests is that the system was built with auditability as a feature requirement, not as a retrofit. The explainability requirement, which regulators required, became a product decision: build a system whose reasoning can be traced, not just a system whose outputs can be measured.
That is the move. The regulatory constraint did not prevent the AI feature. It shaped it. And the feature that shipped was more trustworthy, to regulators and to users, than a less constrained version would have been.
How to Involve Compliance in Discovery Without It Becoming a Veto Relationship
The failure of "involve legal early" as advice is that most product teams interpret it as "invite legal to the first meeting and let them tell you what you cannot build." That produces the same veto, earlier in the process, with less to show for it.
Involving compliance in discovery means something more specific than that.
Start with the user problem, not the solution. When you bring a problem to legal before you have built anything, the conversation is exploratory. "We are trying to help a physician see medication history across multiple visits without requiring them to log into three systems" is a solvable problem. "We built an AI that ingests records from multiple systems and surfaces them in a unified view" is a feature with a compliance gap.
Ask for the constraint, not the verdict. "What does this system need to do, or not do, for it to be compliant?" is a different question than "Is this compliant?" The first question produces requirements. The second question produces a yes or a no.
Translate regulatory requirements into acceptance criteria. When legal tells you that a model must be explainable to satisfy audit requirements, that is not a legal problem. That is a product specification. Write it as one. The model must produce, for each recommendation, a human-readable explanation of the top three factors that influenced the output. That is something engineering can build to.
Treat the first legal conversation as a constraints discovery session, not a review. The output is not approval. The output is a set of non-negotiable constraints and a set of negotiable ones. Non-negotiable constraints go into the product brief. Negotiable constraints become design tradeoffs.
The Judgment Turn
Here is the position that most articles about AI in regulated industries will not take directly:
"Legal said no" is almost always "legal said no to the version we built without asking them."
The veto is real. The feature is dead. But the sequence that produced the veto was a product process failure, not a legal process failure. Legal reviewed what they were given, at the time they were included, and they found something that violated a requirement they had been enforcing for years.
The uncomfortable position is this: if you are a PM in a regulated industry and you are waiting to involve compliance until you have something to show them, you are not moving fast. You are accumulating risk that will surface at the worst possible moment, when the cost of fixing it is highest and the time to launch is shortest.
The three failure modes, shipping nothing, compliance surprise, compliance theater, share a root cause. The team treated the compliance layer as something that happens to their feature, not something they design into it.
The teams that ship AI in regulated industries treat the constraint the same way a good PM treats any hard constraint: as a design input that reveals what the product actually needs to be.
Key Takeaways
- "Legal said no" is usually a sequence problem, the feature was built before legal had input, not because the feature was impossible.
- The three failure modes in regulated AI (shipping nothing, compliance surprise, compliance theater) all share a root cause: compliance was not a design input.
- Asking "what does this need to be true to be compliant?" produces requirements. Asking "is this compliant?" produces a verdict.
- Regulatory constraints, like explainability requirements, can be translated directly into product specifications and acceptance criteria. They are not inherently opposed to good product design.
- Involving compliance at discovery is not about getting early approval. It is about replacing rebuild cycles with informed design.