Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Issues to Insights: RAG-based Explanation Generation from Software Engineering Artifacts

Daniel Pöttgen, Mersedeh Sadeghi, Max Unterbusch, Andreas Vogelsang

TL;DR

This paper demonstrates that retrieval augmented generation using issue tracking data can generate user facing explanations of software behavior with high fidelity. By building a three stage RAG pipeline that indexes, retrieves, and generates explanations grounded in GitHub issues for Mattermost, the approach achieves strong alignment with human references and maintains high faithfulness to retrieved evidence. The study reveals that larger open weight LLMs yield more trustworthy explanations and that retrieval quality is crucial to preventing hallucinations, while noting data quality and privacy considerations as practical limits. Overall, the work presents a scalable, data driven path to explainability in general software systems beyond AI models, leveraging development artifacts that evolve with the codebase.

Abstract

The increasing complexity of modern software systems has made understanding their behavior increasingly challenging, driving the need for explainability to improve transparency and user trust. Traditional documentation is often outdated or incomplete, making it difficult to derive accurate, context-specific explanations. Meanwhile, issue-tracking systems capture rich and continuously updated development knowledge, but their potential for explainability remains untapped. With this work, we are the first to apply a Retrieval-Augmented Generation (RAG) approach for generating explanations from issue-tracking data. Our proof-of-concept system is implemented using open-source tools and language models, demonstrating the feasibility of leveraging structured issue data for explanation generation. Evaluating our approach on an exemplary project's set of GitHub issues, we achieve 90% alignment with human-written explanations. Additionally, our system exhibits strong faithfulness and instruction adherence, ensuring reliable and grounded explanations. These findings suggest that RAG-based methods can extend explainability beyond black-box ML models to a broader range of software systems, provided that issue-tracking data is available - making system behavior more accessible and interpretable.

From Issues to Insights: RAG-based Explanation Generation from Software Engineering Artifacts

TL;DR

This paper demonstrates that retrieval augmented generation using issue tracking data can generate user facing explanations of software behavior with high fidelity. By building a three stage RAG pipeline that indexes, retrieves, and generates explanations grounded in GitHub issues for Mattermost, the approach achieves strong alignment with human references and maintains high faithfulness to retrieved evidence. The study reveals that larger open weight LLMs yield more trustworthy explanations and that retrieval quality is crucial to preventing hallucinations, while noting data quality and privacy considerations as practical limits. Overall, the work presents a scalable, data driven path to explainability in general software systems beyond AI models, leveraging development artifacts that evolve with the codebase.

Abstract

The increasing complexity of modern software systems has made understanding their behavior increasingly challenging, driving the need for explainability to improve transparency and user trust. Traditional documentation is often outdated or incomplete, making it difficult to derive accurate, context-specific explanations. Meanwhile, issue-tracking systems capture rich and continuously updated development knowledge, but their potential for explainability remains untapped. With this work, we are the first to apply a Retrieval-Augmented Generation (RAG) approach for generating explanations from issue-tracking data. Our proof-of-concept system is implemented using open-source tools and language models, demonstrating the feasibility of leveraging structured issue data for explanation generation. Evaluating our approach on an exemplary project's set of GitHub issues, we achieve 90% alignment with human-written explanations. Additionally, our system exhibits strong faithfulness and instruction adherence, ensuring reliable and grounded explanations. These findings suggest that RAG-based methods can extend explainability beyond black-box ML models to a broader range of software systems, provided that issue-tracking data is available - making system behavior more accessible and interpretable.
Paper Structure (14 sections, 2 figures, 1 table)

This paper contains 14 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Proposed Approach and Implementation
  4. RAG Pipeline Architecture
  5. Implementation
  6. Evaluation
  7. LLM-based Evaluation Setup
  8. Evaluation Dimensions
  9. Test Case Design
  10. Results
  11. Discussion
  12. Threats to Validity
  13. Future Work
  14. Conclusion

Figures (2)

  • Figure 1: RAG Architecture Overview
  • Figure 2: Retrieval Process