A 90% Accurate Model That Still Loses Money: Why Churn Prediction Fails Without Uplift Thinking

Recently, I had a small but important reset in how I think about causal inference in real business systems.

For a while, I treated uplift modeling as mostly a statistical problem: define the treatment, derive the causal estimand, estimate the conditional treatment effect, and optimize the targeting policy.

That is technically correct, but incomplete.

In real production environments, especially after models moved into large-scale ranking systems, recommendation systems, and Transformer-based architectures, many traditional causal methods do not automatically translate into stable offline gains. The math may be elegant, but the business impact can still be small, unstable, or hard to scale.

Coupon targeting is a perfect example.

Compared with feed ranking, search ads, or large-scale recommendation, coupon optimization is often a small surface area. But it exposes a much deeper problem:

Many models are accurate at prediction, but wrong about intervention.

This blog explains that gap through a common subscription scenario: a churn model with 90% accuracy that still loses money.

1. The Scenario: A Great Model That Creates Bad Business Results

Imagine a paid community or subscription product launches a churn-prevention model.

The model predicts whether a user is likely to churn. Offline evaluation looks impressive: accuracy is around 90%. When the model predicts that a user is likely to churn, the system triggers a pop-up:

Here is a 50% renewal coupon.

One month later, the team checks the result.

The overall renewal rate increased slightly. Maybe it went up by 0.5%, maybe even 3%. On the surface, this looks like a win.

But finance sees something different.

A large number of users who would have renewed at full price ended up renewing at half price. Renewal count increased, but total renewal revenue dropped.

So the business asks:

If the model is 90% accurate, why is it losing money?

The answer is simple:

The model is solving the wrong problem.

A churn prediction model asks:

Who looks likely to churn?

But the business intervention needs to know:

Who will renew because of this coupon?

Those are two completely different questions.

2. Why 90% Accuracy Can Be a Misleading Number

Let's first talk about accuracy itself.

Suppose this paid community has a natural renewal rate of 70%. That means out of every 100 expiring users, 70 would renew without any intervention.

In that case, even a naive model that predicts everyone will renew already gets 70% accuracy.

So when a churn model reports 90% accuracy, the real lift over the naive baseline is only 20 percentage points.

But even that 20-point improvement does not answer the real business question:

Among the users identified as high risk, how many can actually be saved by a coupon?

A churn model cannot answer that.

It can predict who looks risky. It cannot predict who is persuadable.

That is the core issue.

The model may be accurate at classification, but the business does not need a classification model. It needs an intervention model.

3. Prediction and Intervention Are Not the Same Thing

A traditional churn model predicts:

or equivalently:

But a coupon campaign needs to estimate:

This is the incremental effect of the coupon.

A user may look likely to churn, but a coupon may not change their decision. Another user may not look extremely risky, but a timely discount may persuade them to renew.

This is why a 90% accurate churn model can still destroy ROI.

The model answers:

Who is likely to leave?

But the business needs:

Who will stay because of this intervention?

These two groups overlap much less than most teams assume.

4. Four Types of Users in a Renewal Campaign

To understand the problem clearly, split users by two dimensions:

• Whether they receive the coupon.
• Whether they renew.

This gives us four user types.

1. Natural Renewers

These users will renew whether or not they receive a coupon.

They may be loyal users, heavy users, or people who already planned to renew at full price.

If you give them a coupon, they still renew, but now they pay less.

From the dashboard, they look like successful conversions. From the business perspective, they are margin leakage.

2. Persuadables

These are the users we actually want to find.

They are undecided. Without a coupon, they may not renew. With a coupon, they renew.

This is the true target group for a renewal campaign.

Uplift modeling is mainly about finding this group.

3. Lost Causes

These users will not renew no matter what you do.

They may no longer need the product, may have already moved to another platform, or may have mentally left long before the renewal window.

Sending them a coupon does not create revenue. It only wastes budget and communication cost.

4. Negative Responders

These users may renew without intervention, but the intervention reduces their willingness to pay.

For example, a user may see a sudden discount and think:

Why should I ever pay full price again? I should just wait for coupons.

This group may be small, but in subscription businesses it can create long-term pricing damage.

The key point is:

A churn model tries to find likely churners. An uplift model tries to find persuadable users.

Those are not the same users.

5. A Simple Numerical Example: How the Model Loses Money

Let's make the problem concrete.

Assume a paid community has 10,000 users whose subscriptions are about to expire.

The regular renewal price is $100.

Without any intervention, the natural renewal rate is 70%.

So the baseline revenue is:

Now the team deploys a churn prediction model.

The model identifies 3,000 users as high churn risk and sends all of them a 50% renewal coupon.

That means each renewal from this group now brings only $50 instead of $100.

Inside these 3,000 users, suppose there are three groups.

Now calculate the net impact:

The renewal count increases from 7,000 to 7,300.

The renewal rate improves from 70% to 73%.

But total revenue drops from $700,000 to $640,000.

This is the trap.

The dashboard says the renewal rate improved. The finance team says revenue declined. Both are true.

The model improved the wrong metric.

6. Why This Happens in Real Business

The root cause is not only technical. It is also behavioral.

In many subscription products, users do not decide whether to renew only in the last few days before expiry.

The real decision often forms much earlier: three weeks before expiry, one month before expiry, or gradually throughout the usage cycle.

By the time the final pop-up appears, many users have already made up their minds.

For users who already decided to renew, the coupon only reduces revenue.

For users who already decided to leave, the coupon may be too late.

The only users who are truly affected by the pop-up are those who are still undecided.

These are the persuadable users.

But a churn prediction model usually does not directly identify persuadables. It identifies users who look risky.

This creates a gap between the algorithm team and the product team.

The algorithm team thinks:

High churn risk means pop-up priority.

The product team actually needs:

High persuadability means pop-up priority.

The intersection between these two groups may be much smaller than expected.

7. The Hidden Data Traps

There are several common data traps in this problem.

Trap 1: Average Metrics Hide Structural Loss

A campaign may increase the average renewal rate, but the average hides internal damage.

Among persuadable users, the intervention effect is positive.

Among natural renewers, the intervention effect is negative because the company gives away unnecessary discounts.

When these groups are mixed together, the overall renewal rate may look good while revenue quality worsens.

This is why you cannot only look at renewal rate.

You also need to look at ARPU, total renewal revenue, discount leakage, and incremental ROI.

Trap 2: Self-Selection Bias

Historical data is not neutral.

Users who renewed in the past often have richer behavior data: more logins, more content consumption, more payment history, and more stable engagement patterns.

Users who did not renew often have sparse data.

As a result, the model may become good at identifying active renewers, but weak at distinguishing undecided users from truly lost users.

This is not random noise. It is systematic bias.

Even worse, the bias often appears exactly in the most important group: users near the decision boundary.

Trap 3: Missing Counterfactuals

The most important question is:

What would this user have done if we had not sent the coupon?

But in ordinary historical data, we do not observe both worlds.

For a user who received a coupon and renewed, we do not know whether they renewed because of the coupon or whether they would have renewed anyway.

This missing counterfactual is why churn prediction cannot directly measure intervention value.

Trap 4: Control Groups Are Often Misunderstood

Many product teams resist A/B testing because they worry:

If the control group does not receive a coupon, the user experience will be bad.

But this is often a misunderstanding.

A randomized control group does not mean users receive a worse product. It only means a small group does not receive this specific commercial intervention during the test window.

If you never run this experiment, you will never know whether the coupon creates value or simply burns margin.

Without a control group, every renewal season becomes an expensive guessing game.

8. How to Fix It: Move from Churn Prediction to Uplift Thinking

The correct framework is not:

Who is likely to churn?

The correct framework is:

Who will renew because of the intervention?

The core formula is:

Users with high uplift should receive stronger interventions, such as discount pop-ups.

Users with low or negative uplift should not receive expensive commercial incentives. They may receive lower-cost interventions instead, such as content reminders, product education, in-app messages, or personalized recommendations.

A practical implementation can follow three steps.

Step 1: Run a Small Randomized Experiment

Take a small sample of expiring users.

For example:

• 1,000 users -> treatment group: receive the renewal coupon.
• 1,000 users -> control group: no coupon.

Run this for one renewal cycle and record the renewal outcomes.

This step is often uncomfortable, but necessary.

Without a randomized control group, you cannot separate natural renewal from coupon-driven renewal.

Step 2: Train an Uplift Model

Use the experiment data to train a model that predicts the incremental renewal effect.

The model should not only learn who renews. It should learn whose renewal probability increases when the intervention is applied.

Evaluation should also change.

Do not evaluate this model with ordinary accuracy or AUC alone.

Use metrics such as:

• AUUC: Area Under the Uplift Curve
• Qini coefficient
• Incremental renewal rate
• Incremental ROI

These metrics ask:

If we rank users by predicted uplift, how much incremental renewal can we accumulate?

That is the question the business actually cares about.

Step 3: Design Strategy by Uplift Segment

After scoring users by uplift, the intervention strategy should be segmented.

• High-uplift users receive the strongest intervention, such as a discount pop-up.
• Medium-uplift users receive softer interventions, such as renewal reminders, content highlights, or personalized recommendations.
• Low-uplift users do not receive commercial discounts.
• Negative-uplift users should be protected from aggressive interventions.

This turns the model from a prediction engine into an operating strategy.

9. What Metrics Should Be Monitored?

After moving to uplift thinking, the team should monitor at least two core metrics.

Metric 1: Incremental Renewal ROI

If this number is not positive, the campaign is not creating economic value.

A higher renewal rate alone is not enough.

Metric 2: Natural Renewer Intervention Rate

This measures what percentage of coupon recipients were likely to renew anyway.

If this ratio is too high, for example above 30%, the model is still covering too many users who do not need intervention.

That means the threshold, segmentation, or treatment design needs to be adjusted.

The business should not only ask:

Did renewal rate increase?

It should also ask:

• Did revenue increase?
• Did ARPU decline?
• How much discount was wasted on natural renewers?
• How many truly incremental renewals did we create?
• Are we training users to wait for coupons?

Only then can we judge whether the campaign is truly successful.

10. A Broader Reflection: Uplift Is Not Just a Formula Problem

One important lesson I have learned is that uplift modeling should not be treated as a purely academic exercise.

In papers, we often discuss causal estimators, debiasing methods, treatment effect estimation, and counterfactual learning.

These ideas are valuable, but in production, the harder problem is often not deriving the formula. It is making the system work under real business constraints.

For example:

• The treatment may not be stable.
• The delivery channel may be unreliable.
• The user decision window may be earlier than the intervention window.
• The product team may refuse to hold out a control group.
• The metric may optimize renewal count while silently damaging revenue.
• The target surface may be too small to produce large offline gains.

This is why many causal methods look promising in theory but fail to create stable business improvement offline.

Especially in the era of large models and Transformer-based systems, we should be careful not to believe that every causal modeling problem can scale in the same way as representation learning.

Uplift modeling is useful, but it is often a small and delicate part of a much larger decision system.

The real value is not showing that you know causal formulas. The real value is knowing when the intervention is meaningful, when the data is credible, and when the business metric is actually aligned with profit.

11. What If a Full A/B Test Is Not Possible?

In real companies, full randomization may not always be possible.

Maybe the product team refuses to withhold coupons from a control group. Maybe the campaign is tied to a key revenue period. Maybe leadership wants every high-risk user to receive some intervention.

In that case, observational causal methods can help, but they should be treated as approximations.

Propensity Score Matching

You can match treated users with similar untreated users based on observed features such as activity level, payment history, content usage, membership duration, device type, and historical engagement.

The limitation is that matching only controls for observed variables. It cannot control for hidden intent.

Inverse Probability Weighting

You can estimate the probability that each user receives the intervention and reweight users to make treated and untreated groups more comparable.

The limitation is that this method depends heavily on correctly modeling treatment assignment.

Doubly Robust Estimation

This method combines propensity modeling and outcome modeling.

It is more robust than using only one model, but it still fails if both models are wrong or if important hidden confounders are missing.

Causal Forests and Meta-Learners

Methods such as T-Learner, S-Learner, X-Learner, R-Learner, and causal forests can estimate heterogeneous treatment effects.

They are useful for identifying which types of users respond more strongly to intervention.

But they still require credible treatment and control information. If the historical data is heavily biased, a sophisticated model will simply produce sophisticated bias.

The best practical solution is usually not to avoid experimentation completely.

It is to run a small, carefully designed randomized holdout.

12. A Strong Interview-Style Answer

If I were answering this question in an interview, I would say:

The 90% accuracy model is solving a churn prediction problem, but the business needs to solve an intervention problem. The model can predict who looks likely to churn, but it cannot tell us who will renew because of the coupon. These are different targets.

The campaign loses money because it sends discounts to many natural renewers. These users would have renewed at full price, but after receiving the coupon, they renew at a discount. The renewal rate may increase, but ARPU and total revenue decline.

To fix this, I would move from a churn prediction model to an uplift model. The target should be P(renewal | coupon) - P(renewal | no coupon).

To train this properly, I would run a randomized A/B test with a treatment group and a control group. Then I would evaluate the model using AUUC, Qini coefficient, incremental renewal rate, and incremental ROI instead of only accuracy or AUC.

I would also segment users by predicted uplift. High-uplift users receive the discount. Medium-uplift users receive softer reminders. Low-uplift or negative-uplift users do not receive costly commercial incentives.

Finally, I would monitor natural renewer intervention rate and incremental renewal ROI every cycle. If too many natural renewers are receiving coupons, the campaign is not growth. It is margin leakage.

Final Takeaway

A high-accuracy churn model helps you find users who may leave.

An uplift model helps you find users whose behavior can be changed.

That difference matters because business growth does not come from predicting risk. It comes from creating incremental value.

In subscription and renewal scenarios:

• Churn prediction tells you who looks risky.
• Uplift modeling tells you who is worth saving.

If you confuse the two, the model may improve renewal rate while quietly destroying revenue.