Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust prediction under missingness shifts

Patrick Rockenschaub, Zhicong Xian, Alireza Zamanian, Marta Piperno, Octavia-Andreea Ciora, Elisabeth Pachl, Narges Ahmidi

Abstract

Prediction becomes more challenging with missing covariates. What method is chosen to handle missingness can greatly affect how models perform. In many real-world problems, the best prediction performance is achieved by models that can leverage the informative nature of a value being missing. Yet, the reasons why a covariate goes missing can change once a model is deployed in practice. If such a missingness shift occurs, the conditional probability of a value being missing differs in the target data. Prediction performance in the source data may no longer be a good selection criterion, and approaches that do not rely on informative missingness may be preferable. However, we show that the Bayes predictor remains unchanged by ignorable shifts for which the probability of missingness only depends on observed data. Any consistent estimator of the Bayes predictor may therefore result in robust prediction under those conditions, although we show empirically that different methods appear robust to different types of shifts. If the missingness shift is non-ignorable, the Bayes predictor may change due to the shift. While neither approach recovers the Bayes predictor in this case, we found empirically that disregarding missingness was most beneficial when it was highly informative.

Robust prediction under missingness shifts

Abstract

Prediction becomes more challenging with missing covariates. What method is chosen to handle missingness can greatly affect how models perform. In many real-world problems, the best prediction performance is achieved by models that can leverage the informative nature of a value being missing. Yet, the reasons why a covariate goes missing can change once a model is deployed in practice. If such a missingness shift occurs, the conditional probability of a value being missing differs in the target data. Prediction performance in the source data may no longer be a good selection criterion, and approaches that do not rely on informative missingness may be preferable. However, we show that the Bayes predictor remains unchanged by ignorable shifts for which the probability of missingness only depends on observed data. Any consistent estimator of the Bayes predictor may therefore result in robust prediction under those conditions, although we show empirically that different methods appear robust to different types of shifts. If the missingness shift is non-ignorable, the Bayes predictor may change due to the shift. While neither approach recovers the Bayes predictor in this case, we found empirically that disregarding missingness was most beneficial when it was highly informative.

Paper Structure

This paper contains 51 sections, 3 theorems, 32 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem definition
  4. Robust prediction under missingness shift
  5. Importance of unbiased estimation of $f^\star$ for robust prediction
  6. Complete observations as a special case
  7. Learning under ignorable missingness shifts
  8. Robust end-to-end learning: NeuMISE
  9. The role of Y
  10. Unbiased estimation introduces shifts in Y-dependent missingness
  11. The effect of shifts in Y-dependence
  12. Experiments
  13. Data
  14. Missingness scenarios
  15. Baselines
  16. ...and 36 more sections

Key Result

Theorem 1

Assume that the data is generated according to Equation (eq:data-gen). Then, if there is a missingness shift, ${\forall m \in \{0,1\}^d:}$${\Tilde{f^\star}_m = \Tilde{g^\star}_m}$ holds in general only if the missingness shift is ignorable.

Figures (8)

  • Figure 1: Proposed architecture of NeuMISE compared to the standard NeuMiss architecture.
  • Figure 2: Change in MSE in the target environment with 25% missingness relative to the analytical Bayes predictor. We evaluated estimators trained in the target environment (no shift) and estimators trained in another source environment (shift from 50% to 25%) across 10 random reruns.
  • Figure 3: Change in MSE in simulated data compared to the complete data predictor. Due to dependence on $Y$, performance may be better than on complete data.
  • Figure 4: MSE in the semi-simulated LBIDD data without access to ground truth covariate data.
  • Figure 5: MSE in the target environment (25% missingness) compared to the performance of the Bayes predictor. We evaluated estimators trained within the same environment (no shift) and estimators trained in another source environment (shift from 50% missingness).
  • ...and 3 more figures

Theorems & Definitions (5)

  • Definition 1: Missingness shift
  • Definition 2: Ignorable missingness shift
  • Theorem 1: Equivalence of Bayes predictors under missingness shift
  • Corollary 1: Non-ignorability of Y-dependent missingness shifts
  • Theorem 1: Equivalence of Bayes predictors under missingness shift