Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL

TL;DR

Pharmacovigilance data environments are large, multi-table databases that pose barriers to non-technical users seeking information. The authors propose a retrieval-augmented generation pipeline using GPT-4, augmented by a business context document that encodes domain knowledge and rules to guide NLQ-to-SQL translation. Across three phases, the study shows that contextual knowledge dramatically boosts NLQ-to-SQL accuracy from 8.3% (schema-only) to 78.3% (with context), with further improvements when excluding high-complexity queries. The work highlights the importance of domain-specific context for accurate, context-aware query generation and demonstrates a scalable approach that could democratize PV data access and be extended to other data-intensive domains.

Abstract

Objective: To enhance the efficiency and accuracy of information retrieval from pharmacovigilance (PV) databases by employing Large Language Models (LLMs) to convert natural language queries (NLQs) into Structured Query Language (SQL) queries, leveraging a business context document. Materials and Methods: We utilized OpenAI's GPT-4 model within a retrieval-augmented generation (RAG) framework, enriched with a business context document, to transform NLQs into syntactically precise SQL queries. Each NLQ was presented to the LLM randomly and independently to prevent memorization. The study was conducted in three phases, varying query complexity, and assessing the LLM's performance both with and without the business context document. Results: Our approach significantly improved NLQ-to-SQL accuracy, increasing from 8.3\% with the database schema alone to 78.3\% with the business context document. This enhancement was consistent across low, medium, and high complexity queries, indicating the critical role of contextual knowledge in query generation. Discussion: The integration of a business context document markedly improved the LLM's ability to generate accurate and contextually relevant SQL queries. Performance achieved a maximum of 85\% when high complexity queries are excluded, suggesting promise for routine deployment. Conclusion: This study presents a novel approach to employing LLMs for safety data retrieval and analysis, demonstrating significant advancements in query generation accuracy. The methodology offers a framework applicable to various data-intensive domains, enhancing the accessibility and efficiency of information retrieval for non-technical users.

Figures (5)

• Figure 1: Phase 1 Experimental Design
• Figure 2: Phase 2 Experimental Design
• Figure 3: LLM Chatbot User Interface
• Figure 4: System Architectural Diagram
• Figure 5: Boxplot of SQL Complexity Scores