Investigating the use of Snowballing on Gray Literature Reviews

TL;DR

This paper addresses how to extend snowballing, a standard SLR technique, to gray literature in software engineering by applying forward and backward SB to Stack Exchange discussions about technical debt. Using a start set of 108 TD-related SEPM discussions, the authors perform two SB approaches (link-based and similarity-based) and compare them to the original database-search method. A single SB iteration yields 291 new discussions, with 130 deemed valid, representing about a 120% increase in the data set and similar precision to the original search; similarity-based SB shows strong recall with competitive precision. The study provides a practical framework for applying SB to Q&A platforms, highlights the complementary value of related versus linked links, and demonstrates how GL reviews can be expanded to better capture practitioner perspectives in SWE.

Abstract

Background: The use of gray literature (GL) has grown in software engineering research, especially in studies that consider Questions and Answers (Q&A) sites, since software development professionals widely use them. Though snowballing (SB) techniques are standard in systematic literature reviews, little is known about how to apply them to gray literature reviews. Aims: This paper investigates how to use SB approaches on Q&A sites during gray literature reviews to identify new valid discussions for analysis. Method: In previous studies, we compiled and analyzed a set of Stack Exchange Project Management (SEPM) discussions related to software engineering technical debt (TD). Those studies used a data set consisting of 108 valid discussions extracted from SEPM. Based on this start data set, we perform forward and backward SB using two different approaches: link-based and similarity-based SB. We then compare the precision and recall of those two SB approaches against the search-based approach of the original study. Results: In just one snowballing iteration, the approaches yielded 291 new discussions for analysis, 130 of which were considered valid for our study. That is an increase of about 120% over the original data set (recall). The SB process also yielded a similar rate of valid discussion retrieval when compared to the search-based approach (precision). Conclusion: This paper provides guidelines on how to apply two SB approaches to find new valid discussions for review. To our knowledge, this is the first study that analyzes the use of SB on Q&A websites. By applying SB, it was possible to identify new discussions, significantly increasing the relevant data set for a gray literature review.

Figures (6)

• Figure 1: Example of a discussion.
• Figure 2: Data extraction and filtering process.
• Figure 3: Discussions analysis procedure.
• Figure 4: Example of a valid discussion.
• Figure 5: Example of false positive discussion.