Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA

V Venktesh, Mandeep Rathee, Avishek Anand

TL;DR

SUNAR tackles the bounded-recall problem in open-domain complex QA by leveraging a neighborhood-aware retrieval strategy guided by LLM-derived uncertainty signals. It builds a document neighborhood graph and iteratively expands the retrieval pool with uncertainty-aware re-scoring and neighbor exploration, then applies a post-hoc meta-evidence reasoner to correct reasoning paths. The combination of ASU-based retrieval scoring and MER-based reasoning yields substantial improvements over state-of-the-art retrieve-and-reason baselines on MusiqueQA and 2WikiMultiHopQA, with up to 31.84% performance gains reported. The approach is demonstrated to be robust across LLM substrates and compatible with alternative query-understanding pipelines, highlighting its potential as a general framework for improving recall and reasoning in complex QA systems. Overall, SUNAR provides a principled method to mitigate bounded recall and distractor noise in RAG-style QA while maintaining compatibility with existing retrieval and reasoning components.

Abstract

Complex question-answering (QA) systems face significant challenges in retrieving and reasoning over information that addresses multi-faceted queries. While large language models (LLMs) have advanced the reasoning capabilities of these systems, the bounded-recall problem persists, where procuring all relevant documents in first-stage retrieval remains a challenge. Missing pertinent documents at this stage leads to performance degradation that cannot be remedied in later stages, especially given the limited context windows of LLMs which necessitate high recall at smaller retrieval depths. In this paper, we introduce SUNAR, a novel approach that leverages LLMs to guide a Neighborhood Aware Retrieval process. SUNAR iteratively explores a neighborhood graph of documents, dynamically promoting or penalizing documents based on uncertainty estimates from interim LLM-generated answer candidates. We validate our approach through extensive experiments on two complex QA datasets. Our results show that SUNAR significantly outperforms existing retrieve-and-reason baselines, achieving up to a 31.84% improvement in performance over existing state-of-the-art methods for complex QA.

SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA

TL;DR

SUNAR tackles the bounded-recall problem in open-domain complex QA by leveraging a neighborhood-aware retrieval strategy guided by LLM-derived uncertainty signals. It builds a document neighborhood graph and iteratively expands the retrieval pool with uncertainty-aware re-scoring and neighbor exploration, then applies a post-hoc meta-evidence reasoner to correct reasoning paths. The combination of ASU-based retrieval scoring and MER-based reasoning yields substantial improvements over state-of-the-art retrieve-and-reason baselines on MusiqueQA and 2WikiMultiHopQA, with up to 31.84% performance gains reported. The approach is demonstrated to be robust across LLM substrates and compatible with alternative query-understanding pipelines, highlighting its potential as a general framework for improving recall and reasoning in complex QA systems. Overall, SUNAR provides a principled method to mitigate bounded recall and distractor noise in RAG-style QA while maintaining compatibility with existing retrieval and reasoning components.

Abstract

Complex question-answering (QA) systems face significant challenges in retrieving and reasoning over information that addresses multi-faceted queries. While large language models (LLMs) have advanced the reasoning capabilities of these systems, the bounded-recall problem persists, where procuring all relevant documents in first-stage retrieval remains a challenge. Missing pertinent documents at this stage leads to performance degradation that cannot be remedied in later stages, especially given the limited context windows of LLMs which necessitate high recall at smaller retrieval depths. In this paper, we introduce SUNAR, a novel approach that leverages LLMs to guide a Neighborhood Aware Retrieval process. SUNAR iteratively explores a neighborhood graph of documents, dynamically promoting or penalizing documents based on uncertainty estimates from interim LLM-generated answer candidates. We validate our approach through extensive experiments on two complex QA datasets. Our results show that SUNAR significantly outperforms existing retrieve-and-reason baselines, achieving up to a 31.84% improvement in performance over existing state-of-the-art methods for complex QA.

Paper Structure

This paper contains 35 sections, 4 equations, 10 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Complex Question Answering
  4. Retrieval Approaches
  5. Pipelines for complex QA
  6. Methodology
  7. Neighborhood Aware Retrieval
  8. Neighborhood Graph Construction
  9. The SUNAR Algorithm
  10. LLM feedback for Neighborhood Aware Retrieval
  11. Experimental Setup
  12. Results
  13. Impact of Neighborhood aware retrieval
  14. Significance of LLM guided NAR for complex QA
  15. Sunar with different LLM substrates and query understanding approaches
  16. ...and 20 more sections

Figures (10)

  • Figure 1: The graph represents the comparison general decompose-retrieve paradigm for complex QA (self-ask), and the core approach of this work Sunar in open-domain setup for complex QA on 2WikiMultihopQA (WQA) (left) and MusiqueQA (MQA) (right).
  • Figure 2: Overview of Sunar with Neighborhood Aware Retrieval (NAR) and LLM based feedback.
  • Figure 3: QA performance for different values of l in top-$l$ documents used for reasoning where $l$={1,3,5,7,10
  • Figure 4: Meta-reasoner prompt
  • Figure 5: Example of In-context learning for MusiqueQA (MQA) through self-ask based prompting of LLMs
  • ...and 5 more figures