PromptMind Team at EHRSQL-2024: Improving Reliability of SQL Generation using Ensemble LLMs

TL;DR

The paper tackles reliable Text-to-SQL in healthcare by decomposing the task into SQL generation and SQL validation, leveraging prompt-informed generation, embedding-guided retrieval, and domain-adapted embeddings. It combines in-context learning with retrieval-augmented fine-tuning (RAFT) and an ensemble-based validation stage to improve robustness on the EHRSQL-2024 task using the MIMIC-IV demo database. Results indicate that ensemble validation markedly reduces errors and yields superior reliability, achieving a top placement in the shared task. The methods show promise for transferable application to other domain-specific Text-to-SQL problems that demand accuracy and reliability.

Abstract

This paper presents our approach to the EHRSQL-2024 shared task, which aims to develop a reliable Text-to-SQL system for electronic health records. We propose two approaches that leverage large language models (LLMs) for prompting and fine-tuning to generate EHRSQL queries. In both techniques, we concentrate on bridging the gap between the real-world knowledge on which LLMs are trained and the domain specific knowledge required for the task. The paper provides the results of each approach individually, demonstrating that they achieve high execution accuracy. Additionally, we show that an ensemble approach further enhances generation reliability by reducing errors. This approach secured us 2nd place in the shared task competition. The methodologies outlined in this paper are designed to be transferable to domain-specific Text-to-SQL problems that emphasize both accuracy and reliability.

Figures (2)

• Figure 1: SQL Generation Process
• Figure 2: Individual vs Ensemble Models