Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies

TL;DR

This work presents a solution that incrementally reveals "just enough" of an ontology that is needed to answer a given question in Large Language Models.

Abstract

Ontologies are known to improve the accuracy of Large Language Models (LLMs) when translating natural language queries into a formal query language like SQL or SPARQL. There are two ways to leverage ontologies when working with LLMs. One is to fine-tune the model, i.e., to enhance it with specific domain knowledge. Another is the zero-shot prompting approach, where the ontology is provided as part of the input question. Unfortunately, modern enterprises typically have ontologies that are too large to fit in a prompt due to LLM's token size limitations. We present a solution that incrementally reveals "just enough" of an ontology that is needed to answer a given question.

Figures (1)

• Figure 1: Natural language querying pipeline