Enhancing Knowledge Retrieval with In-Context Learning and Semantic Search through Generative AI

TL;DR

This work addresses domain-specific knowledge retrieval from vast unstructured and structured data sources without costly fine-tuning. It introduces Generative Text Retrieval (GTR) for text and Generative Tabular Text Retrieval (GTR-T) for tables, both leveraging in-context learning and vector databases to retrieve relevant information and generate or execute appropriate queries via LLMs. Empirical evaluation on MSMARCO shows high Rouge-L performance (≈$0.98$) and strong truthfulness, while Spider-based results for GTR-T demonstrate competitive Text2SQL capabilities (EX≈$0.82$, EM≈$0.60$) against state-of-the-art baselines. The approach promises improved efficiency, scalability, and accessibility of advanced AI-driven information retrieval across domains, with robust evaluation metrics (ROUGE, SAS, EM, EX) supporting its effectiveness.

Abstract

Retrieving and extracting knowledge from extensive research documents and large databases presents significant challenges for researchers, students, and professionals in today's information-rich era. Existing retrieval systems, which rely on general-purpose Large Language Models (LLMs), often fail to provide accurate responses to domain-specific inquiries. Additionally, the high cost of pretraining or fine-tuning LLMs for specific domains limits their widespread adoption. To address these limitations, we propose a novel methodology that combines the generative capabilities of LLMs with the fast and accurate retrieval capabilities of vector databases. This advanced retrieval system can efficiently handle both tabular and non-tabular data, understand natural language user queries, and retrieve relevant information without fine-tuning. The developed model, Generative Text Retrieval (GTR), is adaptable to both unstructured and structured data with minor refinement. GTR was evaluated on both manually annotated and public datasets, achieving over 90% accuracy and delivering truthful outputs in 87% of cases. Our model achieved state-of-the-art performance with a Rouge-L F1 score of 0.98 on the MSMARCO dataset. The refined model, Generative Tabular Text Retrieval (GTR-T), demonstrated its efficiency in large database querying, achieving an Execution Accuracy (EX) of 0.82 and an Exact-Set-Match (EM) accuracy of 0.60 on the Spider dataset, using an open-source LLM. These efforts leverage Generative AI and In-Context Learning to enhance human-text interaction and make advanced AI capabilities more accessible. By integrating robust retrieval systems with powerful LLMs, our approach aims to democratize access to sophisticated AI tools, improving the efficiency, accuracy, and scalability of AI-driven information retrieval and database querying.

Figures (5)

• Figure 1: Methodology overview for GTR showing our embedding-based retrieval system using document chunking, vector database storage, and LLM inference for retrieval of relevant textual data and formulation of answer
• Figure 2: Methodology overview for GTR-T highlighting our approach for structured data retrieval, utilizing table embeddings and LLM generated SQL query to efficiently query large databases
• Figure 3: GTR Reponse and Output Length
• Figure 4: GTR Semantic Evaluation
• Figure 5: Performance Metrics of GTR-T Model