Treatment effect: a critique

TL;DR

The paper contrasts two notions of treatment effect—model-based parameters within a statistical model and model-free counterfactual differences—grounding the discussion in the Fisher–Cox lineage and the concern for generalisability. It uses McCullagh's group-action formalisation, together with a fictitious idealisation where both potential outcomes are observable, to ask when a model-free target $\tau_n=\frac{1}{n}\sum_{i=1}^n \mathbb{E}_i(Y_i(1)-Y_i(0))$ recovers the model-based effect. The analysis shows that in simple additive or multiplicative settings the two formulations align, but in more complex or heterogeneous contexts $\tau_n$ becomes unstable and sample-dependent, challenging extrapolation to new populations. The authors argue for preferring stable, interpretable model-based treatment parameters, while acknowledging ongoing developments in conditional validity, influence functions, and related causal-inference methods as complementary approaches to improve robustness and generalisation.

Abstract

Two broad positions within statistics define a treatment effect, on the one hand, as a parameter of a statistical model, and on the other, as an appropriate population-level difference in outcomes or counterfactual outcomes under the different treatment regimes. This short expository paper presents some simple but consequential insights on the two formulations, contrasting the answers under the most favourable fictitious idealisation for the counterfactual framework. These observations clarify the relationship between Fisherian model-based inference and modern counterfactual formulations, and emphasise concerns, raised by Cox and others, regarding the suitability of model-free definitions as targets of inference when scientific conclusions are intended to generalise beyond the observed sample. Parts of the paper are necessarily controversial; we follow Cox (1958a) in not putting these forward in any dogmatic spirit.