DeepEra: A Deep Evidence Reranking Agent for Scientific Retrieval-Augmented Generated Question Answering

TL;DR

DeepEra tackles the reliability gaps in scientific question answering by introducing an agentic reranker that reasons over candidate passages to assess logical relevance and evidential grounding. It integrates intention recognition, relevance assessment, and evidence filtering with summarization to produce a compact, trustworthy evidence set before answer generation. To evaluate robustness to tricky inputs, the authors construct SciRAG-SSLI, a large SSLI dataset combining naturally retrieved passages with logically inconsistent distractors. Empirical results show that DeepEra improves retrieval robustness and answer fidelity compared to a wide range of baselines, especially under SSLI, and does so with notable efficiency, highlighting practical applicability for scientific QA tasks. Overall, this work is among the first to systematically study SSLI failures in two-stage RAG and offers a scalable, interpretable solution that enhances both evidence quality and generation reliability in scientific retrieval-augmented generation.

Abstract

With the rapid growth of scientific literature, scientific question answering (SciQA) has become increasingly critical for exploring and utilizing scientific knowledge. Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating knowledge from external sources, thereby providing credible evidence for scientific question answering. But existing retrieval and reranking methods remain vulnerable to passages that are semantically similar but logically irrelevant, often reducing factual reliability and amplifying hallucinations.To address this challenge, we propose a Deep Evidence Reranking Agent (DeepEra) that integrates step-by-step reasoning, enabling more precise evaluation of candidate passages beyond surface-level semantics. To support systematic evaluation, we construct SciRAG-SSLI (Scientific RAG - Semantically Similar but Logically Irrelevant), a large-scale dataset comprising about 300K SciQA instances across 10 subjects, constructed from 10M scientific corpus. The dataset combines naturally retrieved contexts with systematically generated distractors to test logical robustness and factual grounding. Comprehensive evaluations confirm that our approach achieves superior retrieval performance compared to leading rerankers. To our knowledge, this work is the first to comprehensively study and empirically validate innegligible SSLI issues in two-stage RAG frameworks.

Figures (10)

• Figure 1: The two-stage RAG-LLM pipeline.
• Figure 2: Retrieval performance of embedding models with similar but logically irrelevant passages.
• Figure 3: Dataset construction pipeline. Scientific abstracts and metadata are collected and clustered to retain contextually related documents. Structured information (Method, Result, Significance) is extracted to generate QA pairs. For each question, SSLI contexts are further generated via LLM-guided instructions to create semantically similar but logically irrelevant distractors.
• Figure 4: Overview of DeepEra. DeepEra first performs Intention Recognition to extract structured query representations. Retrieved candidate passages are then scored in the Relevance Assessment stage using an LLM-based function to prioritize mechanistically or causally relevant evidence. Finally, passages exceeding a relevance threshold are filtered and summarized to produce a compact evidence set for downstream answer generation.
• Figure 5: Visualization of the relevance between questions and retrieved contexts across six evaluation tasks, before and after reranking. Shaded regions denote semantically similar but logically irrelevant passages.