Using LLMs in Generating Design Rationale for Software Architecture Decisions

TL;DR

This study addresses the challenge of undocumented design rationale (DR) for software architecture decisions by leveraging large language models (LLMs). It constructs a 100-entry DR dataset from Stack Overflow and GitHub discussions/issues, and evaluates five LLMs under zero-shot, chain-of-thought (CoT), and multi-agent prompting to generate DR, complemented by semi-structured practitioner interviews. Results show LLMs achieve substantial Recall and produce many useful arguments beyond human-ground-truth DR, with LLM-based agents offering the best alignment to expert reasoning and lower mislead rates, at the cost of longer generation and higher coordination. Practitioners view LLM-generated DR as a valuable assistive tool requiring human validation and richer project context, suggesting practical deployment in design comparisons, unfamiliar architectures, and architectural evolution guidance. The work demonstrates the potential of LLM-driven DR generation for improving architecture decision transparency and maintenance, while outlining avenues to enhance context retrieval, knowledge provenance, and visualization of DR outputs.

Abstract

Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Additionally, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.

Figures (5)

• Figure 1: Overview of the research process
• Figure 2: Dataset construction regarding architecture decisions
• Figure 3: A template of the zero-shot method to generate DR
• Figure 4: A template of the CoT method to generate DR
• Figure 5: Overview of collaboration among LLM-based agents to generate DR