SciNetBench: A Relation-Aware Benchmark for Scientific Literature Retrieval Agents

TL;DR

SciNetBench introduces the first relational benchmark for scientific literature retrieval by constructing a large-scale AI paper corpus and a three-tier task taxonomy (ego-centric, pair-wise, path-wise). Across embedding, agentic, and deep-research retrieval methods, the study reveals substantial gaps in current systems' ability to capture scholarly relations, with relation-aware tasks often yielding recall metrics below 20%. The authors also demonstrate downstream value by showing improvements in literature review when relational ground truth is provided, and they publicly release resources to accelerate progress. Overall, the work highlights the necessity of moving beyond semantic similarity toward relational understanding to support robust scientific knowledge synthesis.

Abstract

The rapid development of AI agent has spurred the development of advanced research tools, such as Deep Research. Achieving this require a nuanced understanding of the relations within scientific literature, surpasses the scope of keyword-based or embedding-based retrieval. Existing retrieval agents mainly focus on the content-level similarities and are unable to decode critical relational dynamics, such as identifying corroborating or conflicting studies or tracing technological lineages, all of which are essential for a comprehensive literature review. Consequently, this fundamental limitation often results in a fragmented knowledge structure, misleading sentiment interpretation, and inadequate modeling of collective scientific progress. To investigate relation-aware retrieval more deeply, we propose SciNetBench, the first Scientific Network Relation-aware Benchmark for literature retrieval agents. Constructed from a corpus of over 18 million AI papers, our benchmark systematically evaluates three levels of relations: ego-centric retrieval of papers with novel knowledge structures, pair-wise identification of scholarly relationships, and path-wise reconstruction of scientific evolutionary trajectories. Through extensive evaluation of three categories of retrieval agents, we find that their accuracy on relation-aware retrieval tasks often falls below 20%, revealing a core shortcoming of current retrieval paradigms. Notably, further experiments on the literature review tasks demonstrate that providing agents with relational ground truth leads to a substantial 23.4% performance improvement in the review quality, validating the critical importance of relation-aware retrieval. We publicly release our benchmark at https://anonymous.4open.science/r/SciNetBench/ to support future research on advanced retrieval systems.

Figures (2)

• Figure 1: Importance of Scientific Networks in Literature Retrieval Scenarios
• Figure 2: Evaluation Protocol of Pair-Wise Retrieval.