Common Functional Decompositions Can Mis-attribute Differences in Outcomes Between Populations
Manuel Quintero, William T. Stephenson, Advik Shreekumar, Tamara Broderick
TL;DR
The paper tackles how to decompose differences in mean outcomes between two populations when the conditional mean $\mathbb{E}[Y|X]$ may be nonlinear. It extends the Kitagawa-Oaxaca-Blinder framework to allow nonlinear, additive decompositions $f$ with a generic $\mathcal{L}$, enabling a split into covariate-shift and outcome-with-covariates components. It shows that common nonlinear tools like FANOVA and ALE can misattribute covariate differences to $Y|X$, and provides theoretical characterizations: in the discrete setting, misattribution is avoided only if the decomposition is distribution-invariant (the Jacobian with respect to the input distribution has a rank-1 form, effectively making it constant across $K_X$); for continuous settings they propose a similar conjecture. These results reveal a fundamental limitation in using single-population decompositions to explain two-population differences and motivate developing new distribution-robust, interpretable decomposition methods for policy evaluation and social science questions.
Abstract
In science and social science, we often wish to explain why an outcome is different in two populations. For instance, if a jobs program benefits members of one city more than another, is that due to differences in program participants (particular covariates) or the local labor markets (outcomes given covariates)? The Kitagawa-Oaxaca-Blinder (KOB) decomposition is a standard tool in econometrics that explains the difference in the mean outcome across two populations. However, the KOB decomposition assumes a linear relationship between covariates and outcomes, while the true relationship may be meaningfully nonlinear. Modern machine learning boasts a variety of nonlinear functional decompositions for the relationship between outcomes and covariates in one population. It seems natural to extend the KOB decomposition using these functional decompositions. We observe that a successful extension should not attribute the differences to covariates -- or, respectively, to outcomes given covariates -- if those are the same in the two populations. Unfortunately, we demonstrate that, even in simple examples, two common decompositions -- functional ANOVA and Accumulated Local Effects -- can attribute differences to outcomes given covariates, even when they are identical in two populations. We provide a characterization of when functional ANOVA misattributes, as well as a general property that any discrete decomposition must satisfy to avoid misattribution. We show that if the decomposition is independent of its input distribution, it does not misattribute. We further conjecture that misattribution arises in any reasonable additive decomposition that depends on the distribution of the covariates.