Navigating Data Corruption in Machine Learning: Balancing Quality, Quantity, and Imputation Strategies

TL;DR

This work investigates how data corruption, including missing and noisy data, degrades performance in two distinct ML paradigms: NLP-SL and Signal-RL. It finds that performance follows an exponential-like diminishing-returns model, S = a (1 - e^{-\lambda (1 - p)}) with a dependent on the corruption rate, and that RL is more sensitive to corruption than NLP. The study also analyzes imputation trade-offs via imputation advantage heatmaps, showing that accurate imputation can help but noisy imputation can hurt, with boundary shapes differing by task. It further demonstrates that enlarging datasets provides only partial resilience to corruption; the required data to offset quality losses grows roughly exponentially, and practical rules (e.g., ~30% data being critical for traffic signals) emerge for prioritizing data collection. Collectively, these findings offer guidelines for robustness in preprocessing, imputation, and data acquisition across noisy ML applications, while outlining avenues for extending the work to additional domains and more sophisticated imputation techniques.

Abstract

Data corruption, including missing and noisy data, poses significant challenges in real-world machine learning. This study investigates the effects of data corruption on model performance and explores strategies to mitigate these effects through two experimental setups: supervised learning with NLP tasks (NLP-SL) and deep reinforcement learning for traffic signal optimization (Signal-RL). We analyze the relationship between data corruption levels and model performance, evaluate the effectiveness of data imputation methods, and assess the utility of enlarging datasets to address data corruption. Our results show that model performance under data corruption follows a diminishing return curve, modeled by the exponential function. Missing data, while detrimental, is less harmful than noisy data, which causes severe performance degradation and training instability, particularly in sequential decision-making tasks like Signal-RL. Imputation strategies involve a trade-off: they recover missing information but may introduce noise. Their effectiveness depends on imputation accuracy and corruption ratio. We identify distinct regions in the imputation advantage heatmap, including an "imputation advantageous corner" and an "imputation disadvantageous edge" and classify tasks as "noise-sensitive" or "noise-insensitive" based on their decision boundaries. Furthermore, we find that increasing dataset size mitigates but cannot fully overcome the effects of data corruption. The marginal utility of additional data diminishes as corruption increases. An empirical rule emerges: approximately 30% of the data is critical for determining performance, while the remaining 70% has minimal impact. These findings provide actionable insights into data preprocessing, imputation strategies, and data collection practices, guiding the development of robust machine learning systems in noisy environments.

Figures (11)

• Figure 1: Simulation environment for Signal-RL experiment. The intersection comprises four approaches, each with three lanes. Traffic demand is generated based on a binomial distribution, with a left-turn:through:right-turn ratio of 1:3:2. The environment state is defined by road occupancy. Each simulation step corresponds to one second, and each episode spans one hour. The step-wise reward is defined in Equation \ref{['eq:signal_reward']}.
• Figure 2: Model convergence in two experiments. (a) NLP-SL experiment: The $x$-axis represents training steps, and the $y$-axis shows validation accuracy. (b) Signal-RL experiment: The $x$-axis denotes training episodes, and the $y$-axis indicates episode reward. Training configurations are detailed in Table \ref{['tab:experiment_setup']}.
• Figure 3: Model performance across varying data missing ratios. The $x$-axis indicates the data missing ratio, while the $y$-axis shows the model score—normalized accuracy for NLP-SL and normalized negative queue length for Signal-RL. (a) NLP-SL experiment; (b) Signal-RL experiment. In both cases, model performance initially declines gradually, followed by a steeper drop and a sharp collapse as the missing ratio approaches 1.0. The RL-trained signal outperforms the fixed-timing signal when the data missing ratio is less than 0.8.
• Figure 4: Relationship between model performance and data corruption ratio, fitted by Equation \ref{['eq:data_corruption_curve']}. The $x$-axis represents $(1 - \text{corruption ratio})$, and the $y$-axis shows the model score—normalized accuracy for NLP-SL and normalized negative queue length for Signal-RL. (a) NLP-SL experiment: Curve fitting parameters are $a = 0.475$, $\lambda = 3.517$, with $R^2 = 0.995$. (b) Signal-RL experiment: Curve fitting parameters are $a = 395.8$, $\lambda = 7.493$, with $R^2 = 0.956$. Both experiments exhibit diminishing returns as data quality improves. The fitted curves align well with the function defined in Equation \ref{['eq:data_corruption_curve']}.
• Figure 5: Signal-RL model performance under varying data corruption ratios. (a) Noise insertion: The $x$-axis represents the noise level. Inserting noise is more detrimental than missing data, often leading to unstable model performance. (b) Masked region: The $x$-axis denotes the masking-region ratio. A ratio of $p$ indicates that the farthest $400 \times p$ meters of each lane are hidden from the model, simulating the limited visual range of video-based vehicle detectors. Model performance remains stable even when up to 70% of less critical data are removed.