Relational Database Augmented Large Language Model

TL;DR

This work addresses the limitation of LLMs in handling up-to-date, private, and PHI-like data by augmenting them with external relational database memory. It introduces an LLM-agnostic architecture featuring a database selection memory, a data value memory, and a set of relational databases, linked by a retrieval pipeline and carefully crafted prompts. Through experiments on a composite dataset spanning multiple public sources, the framework improves SQL generation and answer accuracy for questions requiring database access, while also enhancing robustness to value representation variations. The approach offers practical potential for real-world applications where data correctness, timeliness, and privacy are critical, such as virtual assistants and factual QA systems.

Abstract

Large language models (LLMs) excel in many natural language processing (NLP) tasks. However, since LLMs can only incorporate new knowledge through training or supervised fine-tuning processes, they are unsuitable for applications that demand precise, up-to-date, and private information not available in the training corpora. This precise, up-to-date, and private information is typically stored in relational databases. Thus, a promising solution is to augment LLMs with the inclusion of relational databases as external memory. This can ensure the timeliness, correctness, and consistency of data, and assist LLMs in performing complex arithmetic operations beyond their inherent capabilities. However, bridging the gap between LLMs and relational databases is challenging. It requires the awareness of databases and data values stored in databases to select correct databases and issue correct SQL queries. Besides, it is necessary for the external memory to be independent of the LLM to meet the needs of real-world applications. We introduce a novel LLM-agnostic memory architecture comprising a database selection memory, a data value memory, and relational databases. And we design an elegant pipeline to retrieve information from it. Besides, we carefully design the prompts to instruct the LLM to maximize the framework's potential. To evaluate our method, we compose a new dataset with various types of questions. Experimental results show that our framework enables LLMs to effectively answer database-related questions, which is beyond their direct ability.

Figures (6)

• Figure 1: Overview of our framework. The context switch module first determines if additional information is needed. The memory module retrieves the information from a collection of databases. The output generation module returns the final response based on the user input and the retrieved information (if available).
• Figure 2: An example showing the input and output of the context switch module
• Figure 3: An example showing the input and output of the output generation module.
• Figure 4: Illustration of the proposed memory module. The database selection memory returns top K relevant databases. Then the LLM refines the database selection and generates a retrieval plan following our proposed strategy. For each retrieval target, the data value memory returns relevant data values from the database to help generate correct SQL queries. Then information is retrieved using SQL.
• Figure 5: Dataset Composition