Writing Interview Questions for Products That Do Not Always Work the Same Way

When the Product Disagrees With Itself

Start with a product that changes. Not one that is bad - one that is genuinely non-deterministic. An AI writing assistant that produces a strong paragraph on Tuesday and a weak one on Friday for the same prompt. A content moderation system that flags a post today and passes an identical post tomorrow. A recommendation engine that surfaces the right product for one user segment and completely misses another.

Now design a user interview for that product.

The standard playbook breaks immediately. You cannot ask "did the product give you the right answer" because there is no stable right answer to point at. You cannot show a user "the output" and ask for a reaction, because the output you show is one of many possible outputs - and the one you chose reveals more about your judgment than theirs. You are not studying the product. You are studying your own selection bias.

This is where most AI product research quietly falls apart.

The Validation Theater Problem

Here is the uncomfortable position: most AI product research is validation theater.

The team identifies three or four outputs the system produced that they found impressive. They schedule user interviews. They show those outputs. They ask if the outputs are impressive. They report that users find the product impressive. Someone writes "strong user validation" in the product review deck.

That is not research. That is a confirmation ceremony.

The tell is always the same: the research question was designed to answer "did we build something good" rather than "what does the user actually experience when they use this." These sound similar. They produce completely different interview designs.

Research designed to validate will show users the outputs the team is proud of. Research designed to understand will expose users to the distribution of outputs the product actually generates - including the ones that are mediocre, inconsistent, or confusing. The second type of research is harder to schedule, harder to synthesize, and far more likely to produce a finding the team does not want. That is why it rarely gets run.

Why Non-Determinism Changes Question Design

In a deterministic product - a checkout flow, a search result page, a settings screen - the interviewer and the user are looking at the same object. The question "what do you think of this" has a shared referent.

In a non-deterministic product, that shared referent does not exist unless you manufacture it. And when you manufacture it, you introduce a choice: whose experience are we centering, the user's or the researcher's?

Two specific problems compound this.

Trust calibration is invisible in snapshots. AI products require users to develop a working theory of when to trust the output and when to override it. That theory forms over repeated exposure to the product's full output range - including its failures. A user who has only seen the product succeed has no calibrated trust; they have naive confidence. An interview that only shows successful outputs cannot surface trust calibration, because the user has not had the inputs required to develop one.

Leading questions in AI research hide the experience gap. A leading question in a standard usability interview biases the answer. A leading question in an AI product interview can hide whether the user experienced what you think they experienced at all. If you ask "was the AI suggestion helpful here?" you assume the user noticed it was AI-generated, understood what it was suggesting, and processed it as a discrete input. Any of those assumptions can be wrong. The leading question skips over the gap.

The Comparison: What Question Design Actually Does

The progression down this table is not about sophistication. It is about risk tolerance. Observational prompts and distribution prompts require the research team to accept that users might react to the product's worst outputs, not its best. That finding might be accurate, actionable, and politically inconvenient.

Koo and the Content Moderation Lesson

In 2021 and 2022, the Indian microblogging platform Koo was running content moderation at scale, using AI systems to flag posts that potentially violated community standards. The research challenge this kind of product creates is genuine: how do you evaluate whether an AI's moderation decisions match user expectations without either (a) telling users the AI made the decision, which changes the reaction, or (b) asking normative questions like "should this post be removed," which produces answers shaped by the user's general views on moderation rather than their specific reaction to seeing the content.

The research question design that better surfaces behavioral signal is not "should this post be moderated" but "tell me what you would do if you saw this on your feed."

The answers to those two questions tend to differ meaningfully. When users are asked the normative question, they tend to respond with reference to principle - community standards, platform rules, what they think the correct answer should be. When users are asked the behavioral question, they respond with reference to context - whether they follow the poster, whether the topic matters to them, whether the post appears in a context that makes it feel threatening or merely disagreeable.

When an AI moderation system's decisions are evaluated against users' behavioral responses rather than their normative judgments, the two frequently diverge. That divergence is the finding. It cannot surface from a normative question, because normative questions produce normative answers - not behavioral ones.

The lesson is not specific to content moderation. Any AI product that makes a decision on behalf of the user - recommending, flagging, filtering, generating - contains the same gap between what the AI optimized for and what the user would have done given the same information.

How to Design Research That Surfaces Genuine Reactions

Start from the output distribution, not the output highlight reel. Before writing a single interview question, pull a representative sample of actual product outputs - not the ones the team is proud of, but a random draw weighted by frequency. If the product produces mediocre results sixty percent of the time and excellent results forty percent of the time, your research stimulus should reflect that ratio.

Replace evaluation prompts with action prompts. "What do you think of this?" asks the user to perform an evaluation. "What would you do next?" asks the user to continue a behavior. The second question reveals whether the output landed in the user's working model of the product at all. An output the user ignores entirely is the most important finding you will ever get, and it is invisible to evaluation prompts.

Design for the output the user did not expect. The signal you need most is what happens when the product fails, surprises, or contradicts the user's expectation. Those are the moments where trust calibration is revealed. Build at least one session segment explicitly around an output the user is unlikely to have seen before, and watch - do not ask - what they do with it.

The safest interview question for an AI product evaluation is often no question at all. "Show me what you do when you get this result" extracts more than any question you could design, because it puts the weight of interpretation on the user rather than the researcher. The user has to decide what the result means and what to do about it. That decision sequence, observed without prompting, is the closest thing to field behavior you can get in a controlled setting.

Judgment Turn

Here is what the research process is actually being asked to do: confirm the team's bet.

That is not a cynical reading. It is an accurate description of the incentive structure. The team has shipped something. They need to know if it works. They schedule research. Under that pressure, research protocol drifts toward showing the product at its best, asking questions that assume the user will engage with it the way the team intends, and synthesizing findings that confirm the product is working.

The interviewer who resists that drift is not being difficult. They are doing the job. They are the only person in the process who is positioned to see the gap between what the team believes users experience and what users actually do.

That gap is always present in AI products. The output is non-deterministic. The user's trust calibration is built from repeated exposure to the full distribution. The question design that hides the distribution is not protecting the team - it is delaying the moment when the product's real performance becomes visible.

Good research is what makes that moment early enough to matter.

The cost of not running it is not a missed insight. It is a product decision made on false evidence, shipped at scale, and discovered late - when fixing it is expensive and the team is already defending what they built.

Related Articles

• The AI Product Requirements Document, Writing Specs for Systems That Might Be Wrong
• The User Interview That Actually Teaches You Something
• AI as a Research Partner, Accelerating Discovery Without Outsourcing the Judgment