Can Machine Learning Support the Selection of Studies for Systematic Literature Review Updates?

TL;DR

This study assesses whether supervised text classification can assist in selecting studies for SLR updates in Software Engineering. Using two classifiers (Random Forest and Support Vector Machines) trained on data from an original SLR, the authors quantify effectiveness, effort reduction, and potential human-ML collaboration by comparing against a three-human oracle. The results show ML alone yields only modest effectiveness ($F$-score ≈ 0.33) but can reduce manual screening by up to ≈34% with full recall, while human-only reviewer pairs outperform any human-ML pairing. The authors conclude that ML can ease the workload but cannot replace expert reviewers in the initial screening phase, underscoring the continued primacy of human judgment for rigorous SLR updates. Future directions include adaptive thresholds, hybrid approaches, and exploration of large language models to improve reliability and efficiency.

Abstract

[Background] Systematic literature reviews (SLRs) are essential for synthesizing evidence in Software Engineering (SE), but keeping them up-to-date requires substantial effort. Study selection, one of the most labor-intensive steps, involves reviewing numerous studies and requires multiple reviewers to minimize bias and avoid loss of evidence. [Objective] This study aims to evaluate if Machine Learning (ML) text classification models can support reviewers in the study selection for SLR updates. [Method] We reproduce the study selection of an SLR update performed by three SE researchers. We trained two supervised ML models (Random Forest and Support Vector Machines) with different configurations using data from the original SLR. We calculated the study selection effectiveness of the ML models for the SLR update in terms of precision, recall, and F-measure. We also compared the performance of human-ML pairs with human-only pairs when selecting studies. [Results] The ML models achieved a modest F-score of 0.33, which is insufficient for reliable automation. However, we found that such models can reduce the study selection effort by 33.9% without loss of evidence (keeping a 100% recall). Our analysis also showed that the initial screening by pairs of human reviewers produces results that are much better aligned with the final SLR update result. [Conclusion] Based on our results, we conclude that although ML models can help reduce the effort involved in SLR updates, achieving rigorous and reliable outcomes still requires the expertise of experienced human reviewers for the initial screening phase.

Figures (5)

• Figure 1: Data collection process
• Figure 2: Study design pipeline
• Figure 3: RF inclusion probability distribution
• Figure 4: Reviewers assessment distribution.
• Figure 5: Predictions Distribution considering "uncertain" range.