Signal vs Noise, Revisited

Of your first MVP's 100 data points, real signals are 3–5.

The secret of 5 out of 100

One month after Park's first MVP, she sorted the data. She said with a bright voice, "100 signups!" But on closer look, the situation was different. Of 100 signups, 30 were active, and only 5 actually paid. More interesting: those 5 had completely different patterns. They were using the product in a way different from the other 95.

Park was reading "100 came" as the signal — but the real signal was "5 are paying repeatedly." Both are numbers, but they mean completely different things.

The signal from Season 1, re-read

In Season 1 you learned "the moment you turn gut into data, real business begins." That was true then. But preparing the second MVP, there's a next step. Not all data is signal.

North Star Metric (NSM), AARRR, AI-classified feedback — still needed. But there's a thing they miss: signal quality. Not every one of 100 signups gives the same signal. Inside is real signal, and real noise.

Statistician Nate Silver's The Signal and the Noise emphasizes this: the more data, the easier to mistake noise for signal. Solo founders especially over-interpret daily wiggles. "3 signups yesterday, 2 today" — they read variance as trend.

5 fake signal patterns

1. Vanity metrics

100 signups, 1,000 clicks, 300 concurrent users — nice to show. But if 100 signups = 10 active, is that 1,000 clicks from 100 or from 10? If one person clicks 100 times, is that a signal?

Research shows ~73% of early-stage founders rely on vanity metrics and ignore the warning signals that matter.

2. Friend bubble

"Launch on ProductHunt, get 300 signups day one?" Possible. But are those 300 your real users? Many early signups come from "ProductHunt hunters," "tech news subscribers," "dev community members" — an early-adopter network, not necessarily your real target.

3. One-off interaction

"Users 10x'd suddenly yesterday!" Exciting. Back to baseline the next day. Someone shared in a community. That's a one-off event, not a signal. Real signal appears after. "How many of those 10 came back the next day?"

4. Shallow praise

"Perfect!" "Exactly the tool I needed!" "Amazing!" Sweet to hear, but they don't come back. There's a vast gap between "saying it's good" and "paying for it." Praise is emotion; money is signal.

5. Misreading price rejection as market rejection

"Charged users, churn went to 80%." Lowering price barely helped. Missed: maybe price isn't the problem — maybe too few users ever experienced the core feature. If only 10 of 100 reached the core feature, ask only those 10.

4 traits of a real signal

1. Repetition

Yesterday someone came and did nothing, today they worked 30 minutes — one-off. But 10 people coming every Monday and Friday, ~20 min each — that's a pattern, signal.

Cohort analysis shows it. 50 January signups, 30 January active, 20 returned in February (40% retention), 18 in March (90% retention). That repetition is real signal.

2. Money

"I really love it" vs "I'll pay $9.9/month" are different. Park's 5 of 100 paid — look closely: all marketing-team employees, logged in 3+ times a week, used the core feature within 3 days of signup. That's real signal.

3. Referral

30–50% of early growth in Korean SaaS successes comes from existing-user referrals. "I recommended this to my teammate." "We adopted this at our company." When these come spontaneously, the product is solving a real problem.

4. Inverse of churn (retention)

Healthy B2B SaaS is ≤5% monthly churn. Early stage often 50%+. "What did the people who stayed for over a month have in common?" That's your real user, and their behavior pattern is the real signal.

Compress your real signal to one line

Organize everything into a single proposition. In the next post (Courage to Kill), you'll use that proposition to decide "continue or stop."

Park's compressed signal:

"Marketing-team employees log in 3+ times a week, 60% remain active for one month after signup, and 8% of them convert to paid. So our next MVP is 'go deeper into this segment.'"

Lee's case:

"AI dev tools spread by word of mouth. Of 300 ProductHunt signups, 40 (13%) came via referral. Our signal is 'word-of-mouth in tech community,' and the next MVP's marketing leverages that."

Not just numbers — a proposition that turns directly into "what to do next."

Signal classification matrix

• Real signal: 5 people keep converting to paid monthly
• One-off revenue: one big promo → 30 signups → 80% churn
• Interest signal: logs in weekly but no paid conversion yet (worth waiting)
• Noise: users who churn within 2 days of signup

AI signal classification prompt

Below is behavior data for 100 users of our MVP. (paste)

Classify each user as one of:
- Real signal: repetitive and money-converting
- Interest signal: repetitive but not paid yet
- One-off: only came once or paid once
- Noise: churned within 1 week

For each user, provide JSON with:
- Signup source
- Login count
- Time spent
- Paid conversion (yes/no)
- Last login date

And the 3 common traits per category.

In Claude's output, look at the 3 common traits of the "Real signal" group. That's your next hypothesis.

Wrapping up

Now you've found 3–5 real signals in the noise of 100. That's the end of the first-MVP retro. You've learned to read data — now you must decide. Continue, change direction, or stop. The next post is about the bravest decision — the Courage to Kill.

Previous: What the Data Said vs What You Wanted to Hear
Next: The Courage to Kill — Killing Well Becomes the Next Resource

About the characters (Seoyeon Park, Junho Lee)
Characters in this series are fictional personas created by Fail School. Nate Silver's signal theory, cohort analysis, and B2B SaaS retention benchmarks are all based on real research/statistics.

Minchul Kim, CEO of Freeive, Fail School