Data Quality Matters: Quantifying Image Quality Impact on Machine Learning Performance

TL;DR

This work addresses how image quality degradation from compression and virtualization affects ML-based automotive perception tasks. It proposes a four-step framework to quantify image validity, including data preparation with paired reference and modified images, computation of diverse image quality metrics, evaluation of object detection and semantic segmentation performance, and correlation analysis between quality deviations and performance changes. A key finding is that the LPIPS perceptual metric provides the strongest correlation with performance degradation across both detection and segmentation, outperforming traditional metrics such as MSE, PSNR, and SSIM. The framework enables quantifying uncertainty in test data and informing validation pipelines for simulation environments, with future work on confidence intervals and photorealism variation to extend applicability.

Abstract

Precise perception of the environment is essential in highly automated driving systems, which rely on machine learning tasks such as object detection and segmentation. Compression of sensor data is commonly used for data handling, while virtualization is used for hardware-in-the-loop validation. Both methods can alter sensor data and degrade model performance. This necessitates a systematic approach to quantifying image validity. This paper presents a four-step framework to evaluate the impact of image modifications on machine learning tasks. First, a dataset with modified images is prepared to ensure one-to-one matching image pairs, enabling measurement of deviations resulting from compression and virtualization. Second, image deviations are quantified by comparing the effects of compression and virtualization against original camera-based sensor data. Third, the performance of state-of-the-art object detection models is analyzed to determine how altered input data affects perception tasks, including bounding box accuracy and reliability. Finally, a correlation analysis is performed to identify relationships between image quality and model performance. As a result, the LPIPS metric achieves the highest correlation between image deviation and machine learning performance across all evaluated machine learning tasks.

Figures (5)

• Figure 1: Compression (JPEG 5) or virtualization (vKitti 2) can lead to deviating results in machine learning tasks. The object detection (left) shows false positives compared to the reference. Semantic segmentation (right) shows misclassifications. Our approach aims to quantify deviations in image quality in relation to ML-performance.
• Figure 2: Concept to quantify image deviation and their influence on ML tasks according to section \ref{['sec:Problem_Statement']}. It consists of four steps, data preparation, image quality calculation, ML task performance evaluation and correlation analysis.
• Figure 3: Quantification of the impact of image modification on ML performance for object detection. Values approaching one indicate similar performance on the modified and reference image.
• Figure 4: Quantification of the impact of image modification on ML performance for semantic segmentation. Values approaching one indicate similar performance on the modified and reference image.
• Figure 5: Quantification of image quality for the modification methods. The Labels (low), (high), and (1) indicate which values are considered better.

Theorems & Definitions (3)

• Definition 2.1: Image Quality
• Definition 2.2: Image Modification
• Definition 2.3: Validity