ArchISMiner: A Framework for Automatic Mining of Architectural Issue-Solution Pairs from Online Developer Communities

TL;DR

ArchISMiner presents a two-component framework (ArchPI for automatic ARP identification and ArchISPE for extracting architectural issue–solution pairs) to mine architectural knowledge from Stack Overflow. It introduces ArchISPBench as a ground-truth, sentence-level benchmark and demonstrates state-of-the-art performance in ARP detection ($F1$-score up to $0.960$) and issue–solution extraction ($F1$-scores around $0.883$ for issues and $0.894$ for solutions), validated by a practitioner user study. The approach combines contextual embeddings (BERTOverflow), local lexical features (TextCNN), linguistic heuristics, and similarity-based scoring to deliver concise, high-quality ARP content, and it generalizes to other Stack Exchange forums, yielding a large dataset of over 18K ARP pairs. The work provides open replication resources and discusses practical implications for researchers, platform owners, and tool builders, laying groundwork for automated architectural knowledge reuse in software engineering.

Abstract

Stack Overflow (SO), a leading online community forum, is a rich source of software development knowledge. However, locating architectural knowledge, such as architectural solutions remains challenging due to the overwhelming volume of unstructured content and fragmented discussions. Developers must manually sift through posts to find relevant architectural insights, which is time-consuming and error-prone. This study introduces ArchISMiner, a framework for mining architectural knowledge from SO. The framework comprises two complementary components: ArchPI and ArchISPE. ArchPI trains and evaluates multiple models, including conventional ML/DL models, Pre-trained Language Models (PLMs), and Large Language Models (LLMs), and selects the best-performing model to automatically identify Architecture-Related Posts (ARPs) among programming-related discussions. ArchISPE employs an indirect supervised approach that leverages diverse features, including BERT embeddings and local TextCNN features, to extract architectural issue-solution pairs. Our evaluation shows that the best model in ArchPI achieves an F1-score of 0.960 in ARP detection, and ArchISPE outperforms baselines in both SE and NLP fields, achieving F1-scores of 0.883 for architectural issues and 0.894 for solutions. A user study further validated the quality (e.g., relevance and usefulness) of the identified ARPs and the extracted issue-solution pairs. Moreover, we applied ArchISMiner to three additional forums, releasing a dataset of over 18K architectural issue-solution pairs. Overall, ArchISMiner can help architects and developers identify ARPs and extract succinct, relevant, and useful architectural knowledge from developer communities more accurately and efficiently. The replication package of this study has been provided at https://github.com/JeanMusenga/ArchISPE

Figures (14)

• Figure 1: Examples of an ARP and a programming-related post on SO.
• Figure 2: An example of architectural issues and solutions extracted by our proposed approach from the https://stackoverflow.com/q/29644916/12381813
• Figure 3: Overview of the research methodology
• Figure 4: The structure of the experimental setup for identifying ARPs from SO
• Figure 5: ArchISPE architecture for extracting architectural issue-solution pairs from ARPs