A Large Scale Heterogeneous Treatment Effect Estimation Framework and Its Applications of Users' Journey at Snap
Jing Pan, Li Shi, Paul Lo
TL;DR
The paper tackles scalable estimation of heterogeneous treatment effects in a high-velocity, real-world setting by building an industrial framework trained on hundreds of millions of Snapchat users. It combines cross-experiment data using a two-model uplift approach (T-learner) with a distributed, incremental training pipeline built on TensorFlow Extended to produce stable ITE estimates. The authors identify optimal data selection strategies, compare base learners (with Deep & Cross Network performing best), and demonstrate the value of incremental training for production. Applied to user influenceability and user sensitivity to ads, the framework yields substantial improvements in business metrics and engagement, illustrating practical impact for ad targeting and content optimization.
Abstract
Heterogeneous Treatment Effect (HTE) and Conditional Average Treatment Effect (CATE) models relax the assumption that treatment effects are the same for every user. We present a large scale industrial framework for estimating HTE using experimental data from hundreds of millions of Snapchat users. By combining results across many experiments, the framework uncovers latent user characteristics that were previously unmeasurable and produces stable treatment effect estimates at scale. We describe the core components that enabled this system, including experiment selection, base learner design, and incremental training. We also highlight two applications: user influenceability to ads and user sensitivity to ads. An online A/B test using influenceability scores for targeting showed an improvement on key business metrics that is more than six times larger than what is typically considered significant.