PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR

TL;DR

The paper presents an RLVR-based environment for training search agents to answer scientific QA over biomedical literature. It introduces PaperSearchQA (54,907 training samples, 5,000 test samples) and a 16M-abstract PubMed corpus, plus an evaluation setup using BioASQ. Empirically, RLVR-trained agents outperform non-RL baselines on both PaperSearchQA and BioASQ, with notable gains as model size increases, and qualitative analyses reveal planning, pre-retrieval reasoning, and post-retrieval verification behaviors. The work demonstrates the feasibility and challenges of domain-specific RL-enabled search in AI for science and provides scalable data-generation methods for extending to other scientific domains.

Abstract

Search agents are language models (LMs) that reason and search knowledge bases (or the web) to answer questions; recent methods supervise only the final answer accuracy using reinforcement learning with verifiable rewards (RLVR). Most RLVR search agents tackle general-domain QA, which limits their relevance to technical AI systems in science, engineering, and medicine. In this work we propose training agents to search and reason over scientific papers -- this tests technical question-answering, it is directly relevant to real scientists, and the capabilities will be crucial to future AI Scientist systems. Concretely, we release a search corpus of 16 million biomedical paper abstracts and construct a challenging factoid QA dataset called PaperSearchQA with 60k samples answerable from the corpus, along with benchmarks. We train search agents in this environment to outperform non-RL retrieval baselines; we also perform further quantitative analysis and observe interesting agent behaviors like planning, reasoning, and self-verification. Our corpus, datasets, and benchmarks are usable with the popular Search-R1 codebase for RLVR training and released on https://huggingface.co/collections/jmhb/papersearchqa. Finally, our data creation methods are scalable and easily extendable to other scientific domains.

Figures (4)

• Figure 1: Search agents interleave reasoning and retrieval for question answering (QA). We study QA over scientific literature, contributing an environment for training agents with RL with verifiable rewards (RLVR). We release a training dataset of factoid QA (yellow boxes), a retrieval corpus (purple), and benchmarks.
• Figure 2: Left: the ten question-answering categories defined with experts. Right: example question-answer pairs, which are sufficient supervision for RLVR training methods.
• Figure 3: Data generation pipeline process. Left, generating the categories from \ref{['fig:categories']}: LLM summarizes categories from human-written questions in BioASQ krithara2023bioasq; humans brainstorm categories in parallel; humans synthesize both sources into final categories. Right, QA generation: abstracts from PubMed are sampled and passed to an LLM. The LLM s prompted with categories (and other guidance) to generate QAs. A second LLM paraphrases the QAs to limit exact keyword matching.
• Figure 4: Three interesting behaviours that we observe in search agent traces. We bold some words for emphasis. Since traces are long, we abbreviate them, as indicated by '[...]'. These are discussed further in \ref{['sec:results-qualitative']}.