Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Detecting Architectural Decision Violations

Ruoyu Su, Alexander Bakhtin, Noman Ahmad, Matteo Esposito, Valentina Lenarduzzi, Davide Taibi

TL;DR

This work tackles the problem of automatically detecting architectural decision violations captured as ADRs, a task historically hampered by missing documentation and manual review. It adopts a large-scale, multi-model evaluation where a primary LLM screens potential violations and three validators independently assess the reasoning, augmented with retrieval-augmented code snippets to provide concrete evidence. The study analyzes 980 ADRs from 109 open-source projects, reporting substantial cross-model agreement ($κ=0.724$) and strong accuracy for explicit, code-visible decisions (e.g., $>0.90$ for top performers). However, accuracy degrades for infrastructure- or context-dependent ADRs, with the CIA class remaining the hardest to judge, underscoring the need for hybrid human-in-the-loop workflows and improved retrieval and ADR standardization to reliably scale ADR violation detection in practice.

Abstract

Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.

Evaluating Large Language Models for Detecting Architectural Decision Violations

TL;DR

This work tackles the problem of automatically detecting architectural decision violations captured as ADRs, a task historically hampered by missing documentation and manual review. It adopts a large-scale, multi-model evaluation where a primary LLM screens potential violations and three validators independently assess the reasoning, augmented with retrieval-augmented code snippets to provide concrete evidence. The study analyzes 980 ADRs from 109 open-source projects, reporting substantial cross-model agreement () and strong accuracy for explicit, code-visible decisions (e.g., for top performers). However, accuracy degrades for infrastructure- or context-dependent ADRs, with the CIA class remaining the hardest to judge, underscoring the need for hybrid human-in-the-loop workflows and improved retrieval and ADR standardization to reliably scale ADR violation detection in practice.

Abstract

Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.
Paper Structure (28 sections, 3 figures, 9 tables)

This paper contains 28 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Empirical study design
  4. Goal and Research Questions
  5. Context
  6. Data Collection
  7. Project Selection
  8. ADR Selection
  9. LLM Integration, Design and Validation
  10. Data Analysis
  11. LLM Processing Implementation
  12. Agreement in Detecting ADR Violations Among LLMs (RQ$_1$)
  13. Accuracy, precision, and recall of LLMs in detecting Architectural Decision violations (RQ$_{2}$)
  14. Limitations in detecting Architectural Decision and their violations by LLMs (RQ$_{3}$)
  15. Results
  16. ...and 13 more sections

Figures (3)

  • Figure 1: General LLM Processing Implementation
  • Figure 2: Pairwise Jaccard Coefficients (Overall-C/NC/CIA)
  • Figure 3: Pairwise Jaccard Coefficients (C)