RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answering

TL;DR

RuleRAG addresses core limitations of retrieval-augmented generation in knowledge-intensive QA by injecting symbolic rules into retrieval and reasoning. It presents RuleRAG-ICL and RuleRAG-FT to guide retrievers and generators using KG-derived rules, and creates RuleQA, five rule-aware benchmarks, to evaluate rule-guided QA. The results show substantial gains on RuleQA and generalization to existing RAG datasets, demonstrating the approach's effectiveness and robustness. This work provides a practical framework for integrating knowledge-graph rules into RAG, with implications for more transparent and reliable knowledge-intensive QA systems.

Abstract

Retrieval-augmented generation (RAG) has shown promising potential in knowledge intensive question answering (QA). However, existing approaches only consider the query itself, neither specifying the retrieval preferences for the retrievers nor informing the generators of how to refer to the retrieved documents for the answers, which poses a significant challenge to the QA performance. To address these issues, we propose Rule-guided Retrieval-Augmented Generation with LMs, which explicitly introduces rules for in-context learning (RuleRAG-ICL) to guide retrievers to recall related documents in the directions of rules and uniformly guide generators to reason attributed by the same rules. Moreover, most existing RAG datasets were constructed without considering rules and Knowledge Graphs (KGs) are recognized as providing high-quality rules. Therefore, we construct five rule-aware RAG benchmarks for QA, RuleQA, based on KGs to stress the significance of retrieval and reasoning with rules. Experiments on RuleQA demonstrate RuleRAG-ICL improves the retrieval quality of +89.2% in Recall@10 and answer accuracy of +103.1% in Exact Match, and RuleRAG-FT yields more enhancement. In addition, experiments on four existing RAG datasets show RuleRAG is also effective by offering rules in RuleQA to them, further proving the generalization of rule guidance in RuleRAG.

Figures (6)

• Figure 1: (a) Without the help of rules, the current RAG can only retrieve documents about some keywords, rather than the overall semantics of the query, and thus get confused in answering. (b) Guided by the attributable rule $r$, our proposed RuleRAG retrieves logically supportive documents and then reasons the correct answer .
• Figure 2: The framework of our proposed RuleRAG. RuleRAG-ICL relies on in-context learning with the guidance of rules. RuleRAG-FT involves fine-tuning retrievers and generators ahead. (a) The unified RuleRAG inference process. (b) Rule-guided retriever fine-tuning (RGFT-retriever). (c) Rule-guided generator fine-tuning (RGFT-generator).
• Figure 3: The Reacll@k and EM performance of RuleRAG-FT in RuleQA-I with different numbers of retrieved documents and under multiple circumstances: three settings in DPR (DPR, SSFT-DPR and RGFT-DPR), three settings in SimCSE (SimCSE, SSFT-SimCSE and RGFT-SimCSE) and one setting in BM25. Horizontal numbers over the pillars represent EM for bar charts and slanted numbers around the lines represent Recall@k for line charts.
• Figure 4: The EM variation of RuleRAG-FT produces different characteristics due to the varying difficulty of the rules in our constructed five RuleQA benchmarks.
• Figure 5: The EM of generalizing RuleRAG-FT from the source rule bank $\mathcal{R}_{i}$ to the target rule bank $\mathcal{R}_{j}$ , i.e., RuleRAG-FT is trained on $\mathcal{R}_{i}$ and tested on $\mathcal{R}_{j}$. The numbers in ($\mathcal{R}_{i}$, $\mathcal{R}_{j}$) represent the performance gains compared to the baseline Standard RAG tested on $\mathcal{R}_{j}$.