Scientific Paper Retrieval with LLM-Guided Semantic-Based Ranking

TL;DR

SemRank tackles scientific paper retrieval by marrying LLM-guided query understanding with a concept-based semantic index that encodes multi-granular topics and key phrases. It builds an offline semantic index using a topic classifier and LLM-assisted core concept extraction, then performs a single, efficient LLM-guided retrieval step to identify core concepts and re-rank results through concept-based semantic matching. The approach consistently improves base retrievers across multiple datasets, outperforms several LLM-based baselines, and remains CPU-friendly, with extensions that synergize with LLM-based reranking. This framework enables more accurate, scalable, and faithful retrieval in scientific domains by grounding queries and documents in explicit, corpus-derived concepts.

Abstract

Scientific paper retrieval is essential for supporting literature discovery and research. While dense retrieval methods demonstrate effectiveness in general-purpose tasks, they often fail to capture fine-grained scientific concepts that are essential for accurate understanding of scientific queries. Recent studies also use large language models (LLMs) for query understanding; however, these methods often lack grounding in corpus-specific knowledge and may generate unreliable or unfaithful content. To overcome these limitations, we propose SemRank, an effective and efficient paper retrieval framework that combines LLM-guided query understanding with a concept-based semantic index. Each paper is indexed using multi-granular scientific concepts, including general research topics and detailed key phrases. At query time, an LLM identifies core concepts derived from the corpus to explicitly capture the query's information need. These identified concepts enable precise semantic matching, significantly enhancing retrieval accuracy. Experiments show that SemRank consistently improves the performance of various base retrievers, surpasses strong existing LLM-based baselines, and remains highly efficient.

Figures (5)

• Figure 1: An illustrative example from LitSearch with SPECTER-v2 as base retriever. By capturing the scientific concepts for corpus and query, SemRank substantially improves the ranking results.
• Figure 2: Overview of the SemRank framework.
• Figure 3: Parameter analysis on LitSearch by varying $k$, the number of candidate query concepts.
• Figure 4: Prompts given to the LLM for building semantic index.
• Figure 5: Prompts given to the LLM for query core concept identification.