Complaints trickle in while evals pass
AI product · Quick call · Medium · about 4 minutes
Three months after launch, complaints about the AI feature trickle in weekly. The offline evaluation suite still passes cleanly every night. You are a principal product manager with responsibility for quality and no authority over engineering priorities. Nobody has asked you to investigate. Do you open the investigation anyway, and what would the mistake be?
How to answer a quick call: Say the goal and what a good outcome looks like; Say what you are giving up, and why it is worth it; Say what you would do first, and who has to agree; Name the metric and the threshold that would change your mind.
What this question tests
- Diagnosis
- Technical Fluency
Related PM interview questions
- Measure the Netflix recommendation system
- Prompt injection leak ten days before launch
- Acceptance criterion for an AI feature
- Bigger model or fix document splitting
- Personalised AI trailers for Netflix
- Who owns evaluation of the AI feature