If your feature did not survive a holdout, you shipped the novelty spike.
Choosing the Right Experiment, Bandits, Holdouts, and Lift Tests
When a standard A/B test is the wrong tool, and how to choose between multi-armed bandits, holdout groups, and incrementality tests based on your traffic volume and time constraints. A judgment guide for PMs who want to pick the right method, not the impressive-sounding one.
The Setup Most Teams Skip
You have a new feature. You want to know if it works. Someone on your team says "we should run a multi-armed bandit, it is more efficient." Someone else says "we need a holdout group." Your data analyst mentions "incrementality testing" in passing. You ship the feature with a two-week A/B test and call it done.
That is not always wrong. Sometimes it is exactly right. The mistake is not knowing the difference between when it is right and when it is a genuinely bad trade.
flowchart TD
A[Start] --> B{Sample volume?}
B -->|Below 5k MAU| C[Use A/B Test]
B -->|Above 5k MAU| D{Feedback speed?}
D -->|Signal within 24h| E{Risk tolerance?}
D -->|Signal over 48h| F[Use A/B Test]
E -->|Can afford exploration| G[Use Bandit]
E -->|Need clean read| H{Measurement horizon?}
H -->|Short-term delta| C
H -->|Long-term impact| I{Paid channel?}
I -->|Yes, need causation| J[Use Lift Test]
I -->|No, need novelty-free| K[Use Holdout]Each method answers a different question. Deploying the wrong one does not give you a worse answer to your question, it gives you a confident answer to a question you were not asking.
What Each Method Actually Measures
Standard A/B test: Did variant B perform better than variant A during this time window, on this metric, with these users?
Multi-armed bandit: Which variant should we route more traffic to right now, given the reward signals we have received so far?
Holdout group: What is the true long-term impact of this feature, stripped of novelty effects and novelty decay?
Incrementality test (lift test): Would these users have converted, retained, or engaged anyway, or did this intervention actually cause the outcome?
These are four distinct questions. The fact that all four involve a control group and a treatment group does not make them interchangeable.
The Comparison You Need Before You Decide
| Method | Minimum Traffic Required | Time to Decision | What It Measures | What It Cannot Measure | Organizational Overhead |
|---|---|---|---|---|---|
| A/B Test | Low (hundreds to thousands per variant) | Days to weeks | Short-term metric difference between two defined variants | Long-term effects, novelty decay, causation vs. correlation | Low, most teams can run this independently |
| Multi-Armed Bandit | Medium-to-high (needs rapid feedback loop) | Continuous, no fixed end point | Variant performance in real time, with automatic traffic reallocation | Statistical significance in the traditional sense, long-horizon outcomes, interaction effects | High, requires reward function design and active monitoring |
| Holdout Group | High (you are keeping 5-10% of users out permanently) | Months | True long-term behavioral impact after novelty normalizes | Short-term optimization, variant comparison | Medium-to-high, requires discipline not to collapse the holdout early |
| Incrementality / Lift Test | High (you need a clean control population that received no treatment) | Weeks to months | Causal lift attributable to the intervention, not selection effects | Variant comparison, feature-level optimization | Very high, requires geo-splitting, PSA holdouts, or matched pairs |
The Bandit Problem Nobody Talks About
A multi-armed bandit is not a smarter A/B test. It is a real-time allocation algorithm with a reward function at its center. That reward function encodes what you are optimizing for. If the reward function is wrong, the bandit optimizes for the wrong thing, and it does so confidently, automatically, and at scale.
Most product teams that deploy bandits do not write the reward function. They use a default metric, click-through rate, session length, immediate conversion. The algorithm treats that metric as ground truth. If the true goal is 90-day retention and the proxy metric is 7-day activation, the bandit will send 80% of traffic to the variant that looks best on 7-day activation and will never know whether it made retention worse.
Here is the judgment call: A multi-armed bandit deployed without a data scientist reviewing the reward function is not more sophisticated than an A/B test. It is an A/B test where the PM outsourced the stopping decision to an algorithm they cannot explain.
The bandit also has a second structural problem. It requires fast, frequent reward signals to work. If your metric takes three days to surface, a purchase decision, a subscription upgrade, a referral action, the algorithm is allocating traffic based on noise while waiting for the real signal to arrive. In that regime, a fixed-horizon A/B test with a pre-specified sample size is more reliable, not less.
Use a bandit when: the reward signal is immediate (click, play, add-to-cart), you have high traffic volume, you genuinely cannot afford to hold a control at full scale, and a data scientist owns the reward function.
Do not use a bandit because it sounds more rigorous. It is not rigorous by default. It is fast by default.
The Holdout Group as Intellectual Honesty
Spotify's engineering team in Sweden uses holdout groups at the feature level, keeping 5-10% of users out of major feature rollouts for 6-12 months. The reason is not caution. The reason is a specific measurement problem: every feature launch produces a short-term engagement spike from novelty. Users explore new surfaces. They interact with new UI elements out of curiosity. If you measure impact two weeks after launch, you are measuring novelty, not value.
The holdout group is the only clean way to answer the question: "After the novelty wears off, do users who have this feature behave differently from users who do not?"
That question takes months to answer by definition. Most teams do not run holdout groups because they cannot hold organizational patience for the answer. Someone will collapse the holdout early to ship to full population. Someone will argue that keeping 5% of users on an inferior experience is unethical. Both of those objections are legitimate. Neither of them makes the short-term A/B result more valid, they just make the team more comfortable with an incomplete answer.
The uncomfortable position: if your feature did not survive a holdout group, your A/B test measured a novelty spike and you shipped the spike.
When the Sophisticated Method Produces Worse Decisions
Three concrete scenarios where choosing the complex method makes things worse, not better:
Low-traffic products running bandits. If you have 2,000 monthly active users, a bandit will spend most of its budget exploring variants before it has enough signal to exploit. A fixed A/B test with a pre-calculated sample size at your actual traffic volume will reach a decision faster and with more interpretable results. Sophistication is not free, it has a traffic cost.
Short-horizon holdouts. A holdout group run for two weeks is not a holdout group. It is an A/B test where the control group label makes you feel more rigorous. The holdout methodology only works if you hold it long enough for novelty to decay, which is typically 8-12 weeks at minimum, and often longer for habit-forming features. Running a "holdout" for 14 days because it felt more advanced produces a worse decision than a 14-day A/B test, because the holdout framing makes you think you measured long-term impact when you did not.
Incrementality tests on tiny channels. Incrementality testing requires a clean separation between treated and untreated populations large enough to detect signal. If you are running a test on a channel that drives 500 conversions per month, you cannot detect a real lift above the noise floor. You will run the test for months, find a statistically insignificant result, and either incorrectly kill a working channel or incorrectly keep a broken one. A simpler proxy analysis, cohort-level spend versus revenue, with seasonal controls, would have given you a faster and less misleading read.
The Lift Test Claim Problem
Most companies that claim they are running lift tests are not running lift tests.
A real incrementality test requires a holdout population that received no exposure to the intervention, not a lower dose, not a delayed exposure, not a different creative. No exposure. That means either geo-based holdouts (entire markets receive no treatment), PSA (Public Service Announcement) substitution (control group sees a neutral ad instead of your ad), or matched-pair design with clean separation.
What most teams actually run: they look at conversion rates for users who clicked their ad versus users who did not. That is not a lift test. That is a self-selected comparison between users who expressed intent and users who did not. Of course the clickers convert at a higher rate. You have not measured incrementality, you have measured intent.
The observation that grounds this: any attribution model that uses last-click or assisted-click data as its input cannot measure incrementality. Those models allocate credit. They do not measure causation.
The Decision Tree With Honest Constraints
Start here: What is your traffic volume?
Below 5,000 monthly active users on the surface you are testing, use a standard A/B test with a pre-specified sample size. Calculate the minimum detectable effect before you start. Do not run a bandit.
Above 5,000 monthly active users, continue.
What is your reward signal latency?
If the outcome you care about surfaces within 24 hours, a bandit is a legitimate option if a data scientist owns the reward function.
If the outcome takes more than 48 hours, use a fixed-horizon A/B test. The bandit's exploration budget will be wasted.
How long can you hold a holdout?
If your organization can commit 8+ weeks without collapsing the holdout, run one on any feature that could plausibly produce novelty-driven short-term gains. New feeds, redesigned navigation, social features. Hold it for 90 days.
If you cannot hold it, acknowledge that your A/B result is a short-term proxy, not a long-term impact measurement. Label it that way in your readout.
Are you measuring a paid channel?
If yes, and you want to claim the channel drives incremental conversions, you need geo-based holdouts or PSA substitution, minimum traffic that supports 80% power at your expected lift. If you cannot meet that bar, report the channel's attributed conversions and explicitly note that incrementality has not been measured. Do not call it a lift test.
Judgment Turn
The pressure to run sophisticated experiments is real. It comes from leadership wanting to show rigor, from data scientists wanting interesting problems, and from PMs wanting to signal that they understand statistics.
The method that produces the best decision is the one calibrated to your actual traffic, your actual reward signal latency, and your actual organizational patience. A two-week A/B test on a feature that needs a 12-month holdout to be meaningful is not rigorous, it is fast, which is a different thing. A bandit on a low-traffic surface with a slow reward signal is not efficient, it is broken, invisibly.
The cost of the wrong experiment is not a bad result. It is a confident, clean-looking result that answers a question you were not asking, and that result will survive in your team's institutional memory as evidence that the feature works, or does not work, long after anyone can remember how it was measured.
Key Takeaways
- Each experimental method answers a structurally different question. Deploying the wrong one gives you a confident answer to the wrong question.
- A multi-armed bandit requires a correct reward function, fast feedback signals, and high traffic volume. Without all three, a fixed A/B test is more reliable.
- A holdout group only measures long-term impact if held long enough for novelty to decay, typically 8-12 weeks minimum.
- Most teams claiming to run lift tests are measuring attributed conversions, not incremental causation. Real incrementality testing requires clean population separation at scale.
- When your traffic, reward latency, or organizational patience cannot support the sophisticated method, the simple method executed correctly produces a better decision.