Most experimentation is negotiation wearing a lab coat.
A/B Tests Settle Arguments, Not Questions
Most A/B tests are not experiments. They are negotiations with extra steps, and the result was decided in the room first.
Most teams do not run A/B tests to learn anything. They run them to settle an argument that two people have already lost patience with, and the data just gives the winner a formal-sounding name.
The setup is familiar. Someone has an opinion. Someone else disagrees. Nobody wants to be the one who backed down in a meeting, so somebody says the magic phrase: "let us just test it." Everyone exhales. The conflict is now procedural instead of personal. The test will decide, and no one has to be wrong out loud.
But the test was never neutral. By the time it launches, the hypothesis has been written to win, the metric has been chosen to confirm, and the sample size has a way of getting locked in right around the moment the result starts looking the way someone wanted. We call this experimentation. Most of the time it is negotiation wearing a lab coat.
The Tell: Were You Ever Rooting for the Other Side?
There is a single question that separates a real experiment from a dressed-up decision. When did you last run a test where you genuinely did not know which variant you wanted to win? A test where you were rooting for the version you personally believed in, watched it lose, and shipped the control anyway?
For most PMs the honest answer is: almost never. And that is the tell.
Real testing requires that you do not yet have a preferred outcome. The instant you have one, every downstream choice bends toward it. Not through fraud, but through ordinary motivated reasoning. You will defend a clean result that agrees with you and interrogate a clean result that does not. You will find the confounder in the test you wanted to fail and wave past the same confounder in the test you wanted to pass. The bias does not feel like bias from the inside. It feels like rigor.
The test is the receipt, not the reason. Most product decisions are made in the room; the experiment is run afterward to give the decision a paper trail.
Where the Bias Hides
The dishonesty, when it exists, is almost never a fabricated number. It lives in the four design choices nobody writes down.
The hypothesis. "Will moving the button increase clicks?" is a question. "Moving the button will increase clicks" is a position. When the hypothesis is phrased as the thing you are hoping to prove rather than the thing you are trying to find out, the test has already taken a side before a single user sees it.
The metric. Pick the metric that your preferred variant happens to move, and you have rigged the scoreboard before kickoff. The variant you favor lifts click-through; the one you oppose lifts task completion. Whichever you name as the success metric decides the winner, and that choice happened in a Slack thread, not in the data. Picking the metric that tells the story you want is the same mistake as treating a single number as the goal; the harder discipline is reading the whole funnel as a diagnostic instead of a scoreboard.
The stopping point. This is the quietest one. If you can keep checking the dashboard and call the test whenever the line crosses in your favor, you are not measuring an effect. You are waiting for noise to flatter you. Peeking at results and stopping on a good day manufactures significance out of randomness. The math only holds if you commit to the sample size before you look.
The segment. When the headline result disappoints, there is always a slice where it worked: new users on mobile in the second week. Carving out the segment that confirms what you wanted, after the fact, is how a failed test gets reported as a qualified success. It is the inverse of doing honest research before you build, where the point is to find out what is true rather than to confirm what you already decided.
None of these require lying. Each one is a defensible judgment call in isolation. Stacked in service of a result you decided on beforehand, they are how a test gets weaponized.
The Take a Manager Might Dispute
Here is the part your manager may not want to hear: a meaningful share of your A/B tests should not be run at all.
The standard line is that more experimentation is always better, that a strong testing culture is a sign of maturity. I would argue the opposite is often true. A team that tests everything is frequently a team that has stopped being willing to make a judgment call and own it. The test becomes a way to launder responsibility. If the variant wins, the data decided. If it loses, the data decided. Either way, no human has to stand behind a choice.
Some decisions genuinely warrant a test: high traffic, a real and surprising disagreement among informed people, a reversible change, and enough volume to reach significance before the result stops being relevant. Most do not clear that bar. They are low-traffic, low-stakes, or already obvious, and the test exists mostly to delay the moment someone has to commit. Deciding which questions are even worth testing is its own filtering job, closer to using the Double Diamond as an elimination tool than to running more experiments. Running it costs you weeks of engineering and analytics time to produce a receipt for a decision you could have made on Tuesday.
That is the trade a testing-heavy culture rarely prices in. Every test consumes setup, instrumentation, a holdout population, and a window of attention, and a large fraction of them were never going to change anyone's mind.
What to Do With This
You do not need a new framework. You need one habit: before you launch, ask out loud what result would change your decision. If the answer is "nothing would, we are shipping this regardless," then you are not running an experiment, and you should stop pretending otherwise. Make the call, own it, and spend the engineering time on a question you actually do not know the answer to.
And when a test does come back against the thing you believed, notice what you do next. If your first move is to hunt for the segment, the confounder, or the reason the metric was wrong, you have just learned something more valuable than the test was ever going to tell you. You learned how much you wanted to be right, and how little you wanted to find out.
The cost of weaponizing a test is not the wasted week. It is that you slowly lose the ability to be surprised, and an organization that can no longer be surprised by its own data has stopped using data at all.
Related Articles
A stakeholder asks you to A/B test a change you already disagree with. Do you run the test, refuse it, or reframe the question being asked?
Make the call in Reps and see whether your reasoning survives the data.
Make the call