RAG-based Question Answering over Heterogeneous Data and Text

TL;DR

QUASAR presents a RAG-based QA framework capable of answering questions over unstructured text, tables, and knowledge graphs with a unified retrieval and generation pipeline. It introduces a Structured Intent (SI) for question understanding and a Re-Ranking and Filtering (RF) stage, including both GNN- and cross-encoder-based approaches, to prune large evidence pools before feeding a moderate-size LLM (8B-Instruct LLaMA-3.1) for answer generation. Across CompMix, TimeQuestions, and Crag, QUASAR achieves competitive or state-of-the-art performance at a fraction of the computational cost of large GPT models, with evidence from all three modalities contributing to improved accuracy. The work also discusses limitations in evidence recall, trust in sources, and challenges in complex multi-hop or aggregated reasoning, outlining future directions for more robust and scalable heterogeneous QA.

Abstract

This article presents the QUASAR system for question answering over unstructured text, structured tables, and knowledge graphs, with unified treatment of all sources. The system adopts a RAG-based architecture, with a pipeline of evidence retrieval followed by answer generation, with the latter powered by a moderate-sized language model. Additionally and uniquely, QUASAR has components for question understanding, to derive crisper input for evidence retrieval, and for re-ranking and filtering the retrieved evidence before feeding the most informative pieces into the answer generation. Experiments with three different benchmarks demonstrate the high answering quality of our approach, being on par with or better than large GPT models, while keeping the computational cost and energy consumption orders of magnitude lower.

Figures (2)

• Figure 1: Overview of the Quasar system.
• Figure 2: Performance of Quasar on the CompMix dev set with different numbers of evidence.