Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imputation for prediction: beware of diminishing returns

Marine Le Morvan, Gaël Varoquaux

TL;DR

The paper empirically investigates whether high-accuracy imputations meaningfully improve downstream predictive performance. By evaluating 26 model-imputation pipelines across 19 MCAR datasets, it finds that gains in prediction are typically small (often less than 10% of the imputation gain), especially as models become more expressive, when a missingness indicator is included, or when outcomes are nonlinear. The results show that simple imputations can be competitive, that a missingness indicator provides consistent benefits even under MCAR, and that MNAR scenarios generally yield smaller improvements from imputation. The work suggests that practical focus should shift toward modeling strategies and informative missingness encodings rather than pursuing increasingly sophisticated imputations for the sake of prediction.

Abstract

Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 19 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, we also show that the use of the missingness indicator is beneficial to the prediction performance, even in MCAR scenarios. Overall, on real-data with powerful models, improving imputation only has a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.

Imputation for prediction: beware of diminishing returns

TL;DR

The paper empirically investigates whether high-accuracy imputations meaningfully improve downstream predictive performance. By evaluating 26 model-imputation pipelines across 19 MCAR datasets, it finds that gains in prediction are typically small (often less than 10% of the imputation gain), especially as models become more expressive, when a missingness indicator is included, or when outcomes are nonlinear. The results show that simple imputations can be competitive, that a missingness indicator provides consistent benefits even under MCAR, and that MNAR scenarios generally yield smaller improvements from imputation. The work suggests that practical focus should shift toward modeling strategies and informative missingness encodings rather than pursuing increasingly sophisticated imputations for the sake of prediction.

Abstract

Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 19 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, we also show that the use of the missingness indicator is beneficial to the prediction performance, even in MCAR scenarios. Overall, on real-data with powerful models, improving imputation only has a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.
Paper Structure (42 sections, 2 theorems, 8 equations, 34 figures, 4 tables)

This paper contains 42 sections, 2 theorems, 8 equations, 34 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Benchmarks.
  4. Theoretical insights.
  5. Experimental setup.
  6. Imputation methods.
  7. Models:
  8. Native handling of missing values:
  9. The datasets
  10. Evaluation strategy
  11. Computational resources.
  12. Results: Determinants of predictions performance with MCAR missingness
  13. Benefits of sophisticated imputation, the indicator, and XGBoost amid high variance.
  14. A detour through imputation accuracies: how do imputers compare?
  15. Linking imputation accuracy and prediction performances.
  16. ...and 27 more sections

Key Result

Proposition C.1

Let $X_1 \in \mathbb R$ be a random variable, and $\beta \in \mathbb R$ a parameter. Furthermore, define: Then:

Figures (34)

  • Figure 1: Relative prediction performances across datasets for different imputations, predictors, and use of the missingness indicator. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are $R^2\,$ scores on the test set relative to the mean performance across all models for a given dataset and repetition. A value of 0.01 indicates that a given method outperforms the average performance on a given dataset by 0.01 on the $R^2\,$ score. Corresponding critical difference plots in \ref{['fig:cd_20_M', 'fig:cd_50_M']}.
  • Figure 2: Left: Imputer performance for recovery. Performances are given as $R^2\,$ scores for each dataset relative to the mean performance across imputation techniques. A negative value indicates that a method perform worse than the average of other methods. Right: Imputation time.
  • Figure 3: Example fit of prediction performance as a function of imputation accuracy, for the Bike_Sharing_Demand dataset and a missing rate of 50%: on the left using an MLP as predictor, and on the right an MLP with missingness indicator.
  • Figure 4: Effect of the imputation recovery on the prediction performance. We report the slope of the regression line where imputation quality is used to predict prediction performance. A coefficient is marked as significantly greater than zero (circle) if the associated p-value (one-sided T-test) is below 0.05 after Bonferroni correction for multiple testing.
  • Figure 5: Correlation between imputation quality and prediction performance. A correlation close to 1 indicates that the quality of imputations is stronly associated to the quality of predictions, while a correlation close to zero means that the quality of predictions is not linked to the quality of imputations. Each correlation is computed using 40 different imputation/performance pairs, made of 4 imputation methods (mean, iterativeBR, missforest, condexp) repeated 10 times.
  • ...and 29 more figures

Theorems & Definitions (4)

  • Proposition C.1: Link between correlation and effect size.
  • proof
  • Proposition L.1: Achieving a targeted missing rate with probit self-censoring
  • proof