RuAG: Learned-rule-augmented Generation for Large Language Models

TL;DR

A novel framework to automatically distill large volumes of offline data into interpretable first-order logic rules, which are injected into LLMs to boost their reasoning capabilities, and its effectiveness in enhancing LLM's capability over diverse tasks is evaluated.

Abstract

In-context learning (ICL) and Retrieval-Augmented Generation (RAG) have gained attention for their ability to enhance LLMs' reasoning by incorporating external knowledge but suffer from limited contextual window size, leading to insufficient information injection. To this end, we propose a novel framework, RuAG, to automatically distill large volumes of offline data into interpretable first-order logic rules, which are injected into LLMs to boost their reasoning capabilities. Our method begins by formulating the search process relying on LLMs' commonsense, where LLMs automatically define head and body predicates. Then, RuAG applies Monte Carlo Tree Search (MCTS) to address the combinational searching space and efficiently discover logic rules from data. The resulting logic rules are translated into natural language, allowing targeted knowledge injection and seamless integration into LLM prompts for LLM's downstream task reasoning. We evaluate our framework on public and private industrial tasks, including natural language processing, time-series, decision-making, and industrial tasks, demonstrating its effectiveness in enhancing LLM's capability over diverse tasks.

Figures (8)

• Figure 1: Comparison of supervised fine-tuning, in-context learning/retrieval-augmented generation, and our proposed learned-rule-augmented generation ($\textit{RuAG}$), which injects logic knowledge to boost generation while reducing computational cost.
• Figure 2: Illustration of logic rules.
• Figure 3: The framework of our novel learned-rule-augmented generation ($\textit{RuAG}$). $\textit{RuAG}$ automatically compresses large external knowledge into compact logic rules using LLM-aided Monte Carlo Tree Search (MCTS), through three phases: LLM-based Logic Rule Search Formulation, Logic Rule Search with MCTS, and Learned-Rule-Augmented Generation. First, the LLM formulates the MCTS search by defining the target and body predicates. Then we apply MCTS to generate structured first-order logic rules, which are applied to guide generation. Our framework provides an efficient alternative to RAG.
• Figure 4: Case studies on relation extraction, log-based anomaly detction and cooperative game.
• Figure 5: Instruction prompt template for generating relation extraction triples.