Rule-Based Explanations for Retrieval-Augmented LLM Systems

TL;DR

This work introduces the first rule-based framework to explain retrieval-augmented LLM outputs by linking the presence or absence of retrieved sources to model behavior through retention and omission rules. It formalizes a precise rule formulation with a recursive validity notion encoded on a lattice of source combinations and develops two mining algorithms, Mono Rule Miner and Dual Rule Miner, that leverage pruning and dynamic programming to efficiently discover 100% valid rules. Through a RAG case study and HotpotQA-based experiments, the authors demonstrate significant pruning efficiency and practical usefulness for auditing misinformation and provenance, as well as clear advantages of the dual-rule approach in capturing reciprocal evidence. The work lays a foundation for actionable explainability in RAG systems and points to future directions like approximate rules, broader benchmarks, and deployable auditing workflows.

Abstract

If-then rules are widely used to explain machine learning models; e.g., "if employed = no, then loan application = rejected." We present the first proposal to apply rules to explain the emerging class of large language models (LLMs) with retrieval-augmented generation (RAG). Since RAG enables LLM systems to incorporate retrieved information sources at inference time, rules linking the presence or absence of sources can explain output provenance; e.g., "if a Times Higher Education ranking article is retrieved, then the LLM ranks Oxford first." To generate such rules, a brute force approach would probe the LLM with all source combinations and check if the presence or absence of any sources leads to the same output. We propose optimizations to speed up rule generation, inspired by Apriori-like pruning from frequent itemset mining but redefined within the scope of our novel problem. We conclude with qualitative and quantitative experiments demonstrating our solutions' value and efficiency.

Figures (6)

• Figure 1: This diagram illustrates an example where the user asks a RAG system to recommend an effective treatment for Long COVID, given three retrieved sources. The rules generated by our algorithm (formalized in Section \ref{['sec:algorithms']}) indicate that retaining source $s_2$ consistently leads the LLM to recommend treatments known to be ineffective.
• Figure 2: A lattice representing an input set $\mathbf{s}$ where $|\mathbf{s}| = 4$. The node coloring illustrates an example run for the top-down breadth-first search performed by Mono Rule Miner (introduced in Section \ref{['subsec:mono']}), which discovers valid rules (blue with solid border), invalid rules (red with long dashes), and candidates that can be pruned (grey with short dashes).
• Figure 3: A lattice corresponding to the case study presented in Section \ref{['subsec:casestudy']}. Here, the minimal valid rule (lowest valid rule in the lattice, as per Definition \ref{['def:rule_minimality']}) states that "if D2 and D4 are retained, then the LLM produces a response that contains misinformation."
• Figure 4: As more HotpotQA supporting sources are provided to a RAG system, our rule miners can prune larger portions of the lattice.
• Figure 5: The average proportion of input subsets evaluated (as measured over two lattices) by Dual vs. two Mono runs (one for each rule type). Two Mono runs results in many overlapping evaluations (identical LLM inference calls); however, Dual's caching mechanism allows it to avoid evaluating any subset more than once.

Theorems & Definitions (8)

• Definition 1: Rule-Based Explanation
• Definition 2: Rule Support
• Definition 3: Rule Confidence
• Definition 4: Retention Rule
• Definition 5: Omission Rule
• Definition 6: Rule Validity
• Definition 7: Rule Subsumption
• Definition 8: Rule Minimality