When Pattern-by-Pattern Works: Theoretical and Empirical Insights for Logistic Models with Missing Values

TL;DR

This work analyzes logistic regression with missing covariates, proving that Pattern-by-Pattern (PbP) learning can closely approximate Bayes probabilities under Gaussian Pattern Mixture Models (GPMM). It shows a Probit PbP be well-specified, and a logistic PbP that nearly matches Bayes predictors under Gaussian assumptions, with performance across MCAR, MAR, and MNAR settings. Through extensive simulations and real-data experiments, the authors compare PbP to mean imputation, MICE, and SAEM, revealing that Mean.IMP is a strong small-sample baseline, PbP excels with large samples (especially for Gaussian features), and nonlinear imputation (MICE with RF/Y) dominates in non-linear regimes, albeit at higher computational cost. The paper provides practical guidance for choosing missing-data strategies, highlighting the curse-of-dimensionality mitigation in real data and the trade-offs between speed and predictive accuracy.

Abstract

Predicting with missing inputs challenges even parametric models, as parameter estimation alone is insufficient for prediction on incomplete data. While several works study prediction in linear models, we focus on logistic models, where optimal predictors lack closed-form expressions. We prove that a Pattern-by-Pattern strategy (PbP), which learns one logistic model per missingness pattern, accurately approximates Bayes probabilities under a Gaussian Pattern Mixture Model (GPMM). Crucially, this result holds across standard missing data scenarios (MCAR and MAR) and, notably, in Missing Not at Random (MNAR) settings where standard methods often fail. Empirically, we compare PbP against imputation and EM methods across classification, probability estimation, calibration, and inference. Our analysis provides a comprehensive view of logistic regression with missing values. It reveals that mean imputation can be used as baseline for low sample sizes and PbP for large sample sizes, as both methods are fast to train and may have good performances in some settings. The best performances are achieved by non-linear multiple iterative imputation techniques that include the response label (Random Forest MICE with response), which are more computationally expensive.

Figures (15)

• Figure 1: Left: When $X_2 \sim \mathcal{N}(0,s^2)$, the best logistic approximation $\sigma(\alpha + \beta x_1)$ closely matches the true probabilities $\mathbb{E}[\sigma(x_1 + X_2)]$, consistent with Theorem \ref{['th:approx_logistic']}. Right: When $X_2 \sim \text{Exp}(\lambda)-\lambda$, the best logistic approximation deviates significantly from the true probabilities. $s^2 \approx 3.83$ and $\lambda \approx 7.63$ are chosen to maximize the deviation from logistic approximation.
• Figure 2: Performances of selected procedures in terms of Misclassification, Calibration, MSE of $\hat{\beta}$ and MAE from Bayes probabilities. Mean/s.e. are computed over 10 replicates of GPMM-MCAR (see \ref{['sec:methodo_SimA']}).
• Figure 3: Performances of selected procedures in terms of Misclassification, Calibration, MSE of $\hat{\beta}$ and MAE from Bayes probabilities. Mean/s.e. are computed over 10 replicates of GPMM-MNAR (see \ref{['sec:methodo_SimA']}).
• Figure 4: Aggregate results of the simulation GPMM (MCAR). A panel of methods are evaluated on Classification (via misclassification rate), Probability Estimation (via MAE from Bayes probabilities), Calibration (via CORP-MCB) and Inference (via MSE of $\widehat{\beta}$). Mean and standard errors over 10 replicates are displayed. Note that the curves from Mean.IMP.M and 05.IMP.M overlap, as do those from MICE.10.M.IMP and MICE.10.IMP for some metrics.
• Figure 5: Performances of selected procedures in terms of MAE from Bayes probabilities. The results are displayed by missing pattern in the test set (with one missing index: [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1]). Means and standard errors over 10 replicates are displayed. Note that the curves from SAEM and MICE.100.Y.IMP overlap.

Theorems & Definitions (4)

• Theorem 3.3
• Theorem 3.5
• Lemma 1.1
• proof : Proof of \ref{['lem:probit_integral']}