To impute or not to impute: How machine learning modelers treat missing data

TL;DR

Missing data in tabular datasets can bias results and erode trust in analyses. The paper conducts a survey of 70 ML researchers and engineers to assess familiarity with missing-data mechanisms (MCAR, MAR, MNAR) and treatment methods, revealing widespread lack of understanding and subjective decision-making. Key findings show a reliance on simple imputation and considerable variability in approach across scenarios, underscoring the need for better education, standardized reporting, and improved analysis tools. The work argues that improved missing-data literacy and reporting practices can enhance reproducibility, validity, and reliability of data-driven conclusions.

Abstract

Missing data is prevalent in tabular machine learning (ML) models, and different missing data treatment methods can significantly affect ML model training results. However, little is known about how ML researchers and engineers choose missing data treatment methods and what factors affect their choices. To this end, we conducted a survey of 70 ML researchers and engineers. Our results revealed that most participants were not making informed decisions regarding missing data treatment, which could significantly affect the validity of the ML models trained by these researchers. We advocate for better education on missing data, more standardized missing data reporting, and better missing data analysis tools.

Figures (4)

• Figure 1: Prevalence of missing data
• Figure 2: Missing data treatment methods mentioned in free response questions. Some participants mentioned multiple methods, so the percentages do not add up to 100%.
• Figure 3: Familiarity with missing data mechanisms
• Figure 4: Familiarity with missing data treatment methods