What the Data Said vs. What You Wanted to Hear

The real result of your first MVP is in the place you didn't look.

Is "5 paid" a good signal?

It's been 6 months since Seoyeon Park shipped her first MVP. The marketing-column auto-classifier did better than expected. 30 signed up the first month; now 100+. Paid conversion 5% — solid by Season-1-reader standards.

The problem starts here. "Are 5 paying customers a good signal, or a failure signal?" She thought about it for a week, no answer. Friends said "great," online communities said "grow it."

But the data she originally saw and the data she sees now are different. Active users falling weekly, average session 1.5 min, refund rate 15%. She can no longer trust her friends' congratulations or her own anxiety. "What am I supposed to be looking at?"

What we wanted to see: the structure of confirmation bias

Reading first-MVP data looks like science but is actually self-deception theology. We think we look objectively, but we see only what we want to see.

Psychologists call this confirmation bias. Our brains accept info that confirms our beliefs, and automatically filter contradicting info.

For a first MVP, this is more lethal. You've already spent time. The 14 days from Season 1, the late nights, the exhaustion. All in the past, and your brain keeps sending the message "please don't let it have been wasted." This is sunk-cost bias, and the result is the same: good data gets enlarged, bad data gets minimized.

Look at Park. She saw "5 paid." OK, that's data. But there's data she didn't see: weekly declining active users, 1.5-min average session, 15% refund. That contradicts "5 paid." But her brain operated automatically. "5 people decided to pay, so this is a real need. The rest is just early-stage."

Confirmation bias isn't intentional lying. You really see that way. The brain evolved to. So self-discipline alone can't beat it. Only systems can.

Objectifying data: use AI as a mirror

Two choices. First, "recognize bias and think more carefully" — doesn't work. Proven in 1950s experiments. Even when careful, your brain still hunts for confirming evidence.

Second is practical. Re-read the data through a third party. AI is excellent at this. No confirmation bias. It doesn't "congratulate" your 5 paid. No sunk cost. Just sees the numbers.

Park can do this:

Data:
- Total signups 100
- Active users 30 (declining weekly)
- Paid conversions 5 ($9.9/month)
- Average session 1.5 min
- Refund rate 15%
- New signups slowed to under 5/week

Looking at this, do you think this product hit PMF?
What other signals should I watch?

AI would answer:

"5 paid is an early signal, but combined with 15% refund and declining active users, this looks more like 'curiosity purchases' than 'real need.' Real signals are: monthly recurring paid ratio, time spent per feature, and customer retention."

That's the power of AI objectification. Pure data reading without your emotions.

With Claude, use the order: "paste cleanly tabulated data" → "ask for critical feedback" → "request 3 next actions." Don't ask everything at once.

Facing negative signals: real reasons users don't use it

Confirmation bias's other face is looking away from negative signals.

In a first MVP, bad data hides in user behavior. Users who signed up but quit after 3 days. Users who never clicked the feature. Users who said "nice" then never returned. These negative signals don't look like "data." They feel emotional and open to interpretation.

But this is the most important data. As psychologist Rob Fitzpatrick wrote in The Mom Test, "what people don't do is far more honest than what they do."

What Park can do is simple. Meet the churned users and ask directly. "You used this tool for 3 days then stopped — what specifically blocked you?" The answer carries far more information than "5 paid."

Two separations

1. What they said vs what they did
2. Initial interest vs real problem

"Nice tool" tells you nothing. But "I have too many marketing columns and classification is hard, and your tool still doesn't solve it" is a concrete signal.

"Just gather lots of feedback" is the wrong instinct. 5 concrete negative pieces beat 100 vague ones. Chasing quantity over quality strengthens confirmation bias.

Re-interpreting signals: reread first-MVP data

Park now lists all her data again. Using Season-1 retro techniques. Sort each data point into 3 categories:

• Clear signal: proven by actual behavior (paid, clear repeated use)
• Weak signal: initial interest, not yet validated (signups, used once or twice)
• Negative signal: what people didn't do (churn, abandonment, refunds)

Now it's clear. "5 paid" isn't "the product is good" — it's "5 people opened their wallet out of curiosity." The real problem is hidden in the 1.5-min session. Users aren't actually classifying columns with this tool.

Signal vs self-rationalization, side-by-side

10 failure-interview questions

For churned or refunded users, the questions to ask:

1. When did you decide to stop using this tool?
2. What specifically got blocked when you quit? (require a clear reason)
3. What did you do instead to solve this problem? (alternative)
4. What problem did you want to solve when you signed up? (original hypothesis)
5. How often does this problem come up? (severity)
6. How are you solving this today? (existing workflow)
7. What feature would have kept you on board? (talk vs real want)
8. Have you used similar tools? What was different? (vs competitor)
9. Any chance you'd try this again? What would need to change? (return condition)
10. Would you recommend this to a friend? (final verdict)

These 10 questions make your next decision far more accurate than "5 paid."

Wrapping up

Now you can tell what your data is saying from what you wanted to hear. 5 paid isn't a good signal — it's just a signal that needs validation. The real signal hides in the unseen, in what people did not do.

Next post: a frame for extracting 3–5 real signals from the 100 unclear data points.

Previous: What Did You Learn?
Next: Signal vs Noise, Revisited

About Seoyeon Park
Seoyeon Park is a fictional persona created by Fail School. Mom Test, confirmation bias, sunk-cost concepts and research are real.

Minchul Kim, CEO of Freeive, Fail School