Identifying Causes of Test Unfairness: Manipulability and Separability
Youmi Suk, Weicong Lyu
TL;DR
This work delivers a causal, interventionist reframing of differential item functioning by decomposing non-manipulable group statuses into manipulable components that operate along distinct causal routes. It defines simple and general separable DIF estimands, derives nonparametric identification under structured assumptions, and proposes practical detection methods using causal forests and BART. Through motivating examples with SAT and Regents items, the paper illustrates how separable-DIF can attribute item bias to construct-irrelevant factors such as sports familiarity or vocabulary barriers, guiding targeted item revisions. The approach advances educational testing by enabling actionable, intervention-based reasoning about fairness, with robust empirical support from simulations and concrete guidance for practitioners. The framework also opens avenues for future work on anchor-item ambiguity, alternative causal methods, and robustness to unmeasured confounding in DIF analysis.
Abstract
Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English vocabulary unfamiliarity and classroom learning barriers, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.