When the Average Treatment Effect Hides Who Actually Benefits
I was working with the Criteo uplift dataset, a randomized advertising experiment with roughly 14 million observations, and the first result looked fairly ordinary. People who were shown an ad visited the advertiser's site about 1.03 percentage points more often than people who were not shown one.
That is a real effect, but it is not a very exciting one. If I only reported the average, the conclusion would basically be that advertising helps a little. What interested me more was whether that average was hiding very different responses across people.
Looking past the average
Because treatment in this dataset was randomized, I already had the important ingredient for estimating a causal effect. The treated and control groups were created by the experiment itself. I still ran a propensity-score matching analysis as a robustness check. After matching similar treated and control observations and checking covariate balance with standardized mean differences, I got an estimated effect of about +0.82 percentage points. The estimate shifted modestly, but the basic conclusion did not change. On average, the intervention helped.
The more interesting question was whether "on average" was masking something useful. So I trained a T-Learner, using separate gradient-boosted models for the treatment and control groups, and estimated how the treatment effect varied across individuals. That changed the picture quite a bit.
The users ranked in the top responder quartile had an estimated uplift of about +3.01 percentage points. At the other end, the lowest-ranked group had an estimated effect of about -0.16 percentage points. So the average of roughly one percentage point was combining people who responded strongly with people for whom the model estimated little benefit, or even slightly negative uplift. That was the result I found most useful.
Why I cared about this result
An average treatment effect answers one question: does the intervention work overall? But in many real systems that is not enough. If the cost of an intervention is meaningful, you also want to know who is actually likely to benefit from it.
In this experiment, treating everyone based on the average effect would throw away a lot of information. The treatment-effect model suggests a more selective approach, concentrating exposure on people predicted to respond strongly while avoiding groups where the model predicts little or negative incremental benefit.
I evaluated that ranking out of sample using the Qini coefficient, which came out to 0.088. I would not read that number in isolation as "good" or "bad." What matters here is whether the ranking actually separates higher-response users from lower-response users on data the model did not train on, rather than just fitting the training set.
One thing I would be careful about
There is an important methodological detail in this analysis. This was already a randomized experiment, so the causal identification comes from the randomization, not from propensity-score matching. I used matching to inspect balance and to check whether the headline estimate stayed reasonable under a different comparison strategy. I would not use this example to claim that matching created the causal effect.
The individual treatment effects also come from a model, so the quartile numbers depend on how well the T-Learner generalizes. That is why I looked at the ranking on held-out observations rather than trusting the training predictions. And this is an advertising experiment, so the approach transfers to other problems but the actual numbers do not.
What I took away from it
The most useful part of this analysis was not the average effect itself. It was seeing how much that average hid.
That is what makes uplift modeling useful to me. Instead of stopping at "does this work?", it lets you ask the more practical question: "who does it work for?"
Analysis based on the public Criteo Uplift Modeling Dataset (Diemert et al., 2018). Treatment assignment in the dataset is randomized; matching here was a robustness and balance check.