Improving Data Cleaning Using Discrete Optimization

TL;DR

This work tackles the challenge of cleaning data with missing entries by focusing on the extreme no-missing-data scenario ($γ=0$) and showing how to reformulate the Mr. Clean IPs as faster linear programs. It introduces a RowCol LP reformulation and a partitioned MaxCol IP, along with a NoMiss Greedy algorithm, to achieve strong data retention with scalable runtimes. The study provides comprehensive empirical comparisons against existing deletion methods, demonstrating superior data retention and practical runtime performance across diverse real-world datasets. The results offer actionable recommendations for large-scale data cleaning and outline avenues for distributed optimization and the downstream effects on imputation.

Abstract

One of the most important processing steps in any analysis pipeline is handling missing data. Traditional approaches simply delete any sample or feature with missing elements. Recent imputation methods replace missing data based on assumed relationships between observed data and the missing elements. However, there is a largely under-explored alternative amid these extremes. Partial deletion approaches remove excessive amounts of missing data, as defined by the user. They can be used in place of traditional deletion or as a precursor to imputation. In this manuscript, we expand upon the Mr. Clean suite of algorithms, focusing on the scenario where all missing data is removed. We show that the RowCol Integer Program can be recast as a Linear Program, thereby reducing runtime. Additionally, the Element Integer Program can be reformulated to reduce the number of variables and allow for high levels of parallelization. Using real-world data sets from genetic, gene expression, and single cell RNA-seq experiments we demonstrate that our algorithms outperform existing deletion techniques over several missingness values, balancing runtime and data retention. Our combined greedy algorithm retains the maximum number of valid elements in 126 of 150 scenarios and stays within 1\% of maximum in 23 of the remaining experiments. The reformulated Element IP complements the greedy algorithm when removing all missing data, boasting a reduced runtime and increase in valid elements in larger data sets, over its generic counterpart. These two programs greatly increase the amount of valid data retained over traditional deletion techniques and further improve on existing partial deletion algorithms.

Figures (6)

• Figure 1: Comparison of Mr. Clean and NoMiss Greedy Algorithms. (a) Percentage of Original Valid Elements Retained (b) Run Time. (c) Percentage of Original Rows Retained. (d) Percent of Original Columns Retained.
• Figure 2: Run Time of RowCol IP and RowCol LP. The green line indicates the maximum allowed run time of 18000 seconds.
• Figure 3: Run Time of MaxCol MIP and Element IP. The green line indicates the maximum allowed run time of 18000 seconds.
• Figure 4: Comparison of Deletion Algorithms at $\gamma=0.0$. (a) Percentage of Original Valid Elements Retained (b) Run Time. (c) Percentage of Original Rows Retained. (d) Percent of Original Columns Retained.
• Figure 5: Comparison of Deletion Algorithms at $\gamma=0.05$. (a) Percentage of Original Valid Elements Retained (b) Run Time. (c) Percentage of Original Rows Retained. (d) Percent of Original Columns Retained.