Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Quality Antipatterns for Software Analytics

Aaditya Bhatia, Dayi Lin, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan

TL;DR

This study develops a taxonomy of ML-specific data quality antipatterns and empirically investigates their prevalence and impact in software defect prediction. Using a SDP dataset with 65 metrics across nine projects, the authors detect eight antipattern types (14 sub-types), assess cleaning order effects, and measure performance and interpretation changes across four learners. They find that tailed distributions, mislabels, class overlap, class imbalance, and correlation & redundancy significantly influence performance in some settings, with neural networks showing the most robustness to cleaning order. Additionally, model interpretations are moderately affected by data quality, underscoring the importance of data-aware evaluation in software analytics. The work provides actionable guidance on antipattern detection, cleaning strategies, and the value of a formal taxonomy to guide SDP practitioners and researchers.

Abstract

Background: Data quality is vital in software analytics, particularly for machine learning (ML) applications like software defect prediction (SDP). Despite the widespread use of ML in software engineering, the effect of data quality antipatterns on these models remains underexplored. Objective: This study develops a taxonomy of ML-specific data quality antipatterns and assesses their impact on software analytics models' performance and interpretation. Methods: We identified eight types and 14 sub-types of ML-specific data quality antipatterns through a literature review. We conducted experiments to determine the prevalence of these antipatterns in SDP data (RQ1), assess how cleaning order affects model performance (RQ2), evaluate the impact of antipattern removal on performance (RQ3), and examine the consistency of interpretation from models built with different antipatterns (RQ4). Results: In our SDP case study, we identified nine antipatterns. Over 90% of these overlapped at both row and column levels, complicating cleaning prioritization and risking excessive data removal. The order of cleaning significantly impacts ML model performance, with neural networks being more resilient to cleaning order changes than simpler models like logistic regression. Antipatterns such as Tailed Distributions and Class Overlap show a statistically significant correlation with performance metrics when other antipatterns are cleaned. Models built with different antipatterns showed moderate consistency in interpretation results. Conclusion: The cleaning order of different antipatterns impacts ML model performance. Five antipatterns have a statistically significant correlation with model performance when others are cleaned. Additionally, model interpretation is moderately affected by different data quality antipatterns.

Data Quality Antipatterns for Software Analytics

TL;DR

This study develops a taxonomy of ML-specific data quality antipatterns and empirically investigates their prevalence and impact in software defect prediction. Using a SDP dataset with 65 metrics across nine projects, the authors detect eight antipattern types (14 sub-types), assess cleaning order effects, and measure performance and interpretation changes across four learners. They find that tailed distributions, mislabels, class overlap, class imbalance, and correlation & redundancy significantly influence performance in some settings, with neural networks showing the most robustness to cleaning order. Additionally, model interpretations are moderately affected by data quality, underscoring the importance of data-aware evaluation in software analytics. The work provides actionable guidance on antipattern detection, cleaning strategies, and the value of a formal taxonomy to guide SDP practitioners and researchers.

Abstract

Background: Data quality is vital in software analytics, particularly for machine learning (ML) applications like software defect prediction (SDP). Despite the widespread use of ML in software engineering, the effect of data quality antipatterns on these models remains underexplored. Objective: This study develops a taxonomy of ML-specific data quality antipatterns and assesses their impact on software analytics models' performance and interpretation. Methods: We identified eight types and 14 sub-types of ML-specific data quality antipatterns through a literature review. We conducted experiments to determine the prevalence of these antipatterns in SDP data (RQ1), assess how cleaning order affects model performance (RQ2), evaluate the impact of antipattern removal on performance (RQ3), and examine the consistency of interpretation from models built with different antipatterns (RQ4). Results: In our SDP case study, we identified nine antipatterns. Over 90% of these overlapped at both row and column levels, complicating cleaning prioritization and risking excessive data removal. The order of cleaning significantly impacts ML model performance, with neural networks being more resilient to cleaning order changes than simpler models like logistic regression. Antipatterns such as Tailed Distributions and Class Overlap show a statistically significant correlation with performance metrics when other antipatterns are cleaned. Models built with different antipatterns showed moderate consistency in interpretation results. Conclusion: The cleaning order of different antipatterns impacts ML model performance. Five antipatterns have a statistically significant correlation with model performance when others are cleaned. Additionally, model interpretation is moderately affected by different data quality antipatterns.
Paper Structure (63 sections, 15 figures, 6 tables)

This paper contains 63 sections, 15 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Data quality issues in general.
  4. Data Quality in Software Analytics.
  5. Taxonomy of ML-specific Data Quality Antipatterns
  6. Resulting Taxonomy
  7. Antipattern Detection and Mitigation Strategies
  8. Packaging Antipatterns
  9. Schema Violations
  10. Data Miscoding
  11. Row Feature Imbalance
  12. Data Distribution Antipatterns
  13. Distribution Shift
  14. Outliers
  15. Skewness
  16. ...and 48 more sections

Figures (15)

  • Figure 1: A taxonomy of data quality antipatterns in structured data.
  • Figure 2: Row feature imbalance. While each project version has 65 columns, the row size differs.
  • Figure 3: Distribution of % Schema violations amongst the different studied project versions of the SDP data.
  • Figure 4: Tailed metrics (left) along with Unnormalized metrics (right) across different project versions of the SDP data.
  • Figure 5: Percentage of duplicated rows detected in the studied SDP dataset.
  • ...and 10 more figures