Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain

Yuyang Li, Philip J. M. Kerbusch, Raimon H. R. Pruim, Tobias Käfer

TL;DR

The paper evaluates three Retrieval-Augmented Generation strategies—Traditional RAG, SQL RAG, and Graph RAG—for a conversational AI system in the airport domain. Using a flight data dataset and two question sets (straightforward and complicated/ambiguous), Graph RAG achieves the highest accuracy (EX) and excels at reasoning, while Traditional RAG delivers strong performance but suffers from hallucinations. SQL RAG also reduces hallucinations by translating NLQ into SQL queries, offering a safer alternative. The study emphasizes question classification and prompt engineering to handle jargon and incomplete queries, highlighting Graph RAG and SQL RAG as preferable for dynamic, safety-critical airport operations.

Abstract

Airports from the top 20 in terms of annual passengers are highly dynamic environments with thousands of flights daily, and they aim to increase the degree of automation. To contribute to this, we implemented a Conversational AI system that enables staff in an airport to communicate with flight information systems. This system not only answers standard airport queries but also resolves airport terminology, jargon, abbreviations, and dynamic questions involving reasoning. In this paper, we built three different Retrieval-Augmented Generation (RAG) methods, including traditional RAG, SQL RAG, and Knowledge Graph-based RAG (Graph RAG). Experiments showed that traditional RAG achieved 84.84% accuracy using BM25 + GPT-4 but occasionally produced hallucinations, which is risky to airport safety. In contrast, SQL RAG and Graph RAG achieved 80.85% and 91.49% accuracy respectively, with significantly fewer hallucinations. Moreover, Graph RAG was especially effective for questions that involved reasoning. Based on our observations, we thus recommend SQL RAG and Graph RAG are better for airport environments, due to fewer hallucinations and the ability to handle dynamic questions.

Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain

TL;DR

The paper evaluates three Retrieval-Augmented Generation strategies—Traditional RAG, SQL RAG, and Graph RAG—for a conversational AI system in the airport domain. Using a flight data dataset and two question sets (straightforward and complicated/ambiguous), Graph RAG achieves the highest accuracy (EX) and excels at reasoning, while Traditional RAG delivers strong performance but suffers from hallucinations. SQL RAG also reduces hallucinations by translating NLQ into SQL queries, offering a safer alternative. The study emphasizes question classification and prompt engineering to handle jargon and incomplete queries, highlighting Graph RAG and SQL RAG as preferable for dynamic, safety-critical airport operations.

Abstract

Airports from the top 20 in terms of annual passengers are highly dynamic environments with thousands of flights daily, and they aim to increase the degree of automation. To contribute to this, we implemented a Conversational AI system that enables staff in an airport to communicate with flight information systems. This system not only answers standard airport queries but also resolves airport terminology, jargon, abbreviations, and dynamic questions involving reasoning. In this paper, we built three different Retrieval-Augmented Generation (RAG) methods, including traditional RAG, SQL RAG, and Knowledge Graph-based RAG (Graph RAG). Experiments showed that traditional RAG achieved 84.84% accuracy using BM25 + GPT-4 but occasionally produced hallucinations, which is risky to airport safety. In contrast, SQL RAG and Graph RAG achieved 80.85% and 91.49% accuracy respectively, with significantly fewer hallucinations. Moreover, Graph RAG was especially effective for questions that involved reasoning. Based on our observations, we thus recommend SQL RAG and Graph RAG are better for airport environments, due to fewer hallucinations and the ability to handle dynamic questions.
Paper Structure (24 sections, 16 figures, 5 tables)

This paper contains 24 sections, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional RAG
  4. SQL RAG
  5. Graph RAG
  6. Dataset and Questions
  7. Experiments
  8. Results
  9. RQ1: How to handle flight data for different questions?
  10. Straightforward questions
  11. Specialized airport jargon, abbreviations, and incomplete questions
  12. Dynamic questions
  13. RQ2: How to reduce hallucinations?
  14. Conclusion
  15. Future Work
  16. ...and 9 more sections

Figures (16)

  • Figure 1: Traditional RAG Method
  • Figure 2: SQL RAG Method
  • Figure 3: Graph RAG Method
  • Figure 4: Method on Ambiguous question dataset
  • Figure 5: Confusion Matrix of Question Classifications
  • ...and 11 more figures