PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization

TL;DR

PaperScout reframes scholarly search as a sequential decision process, addressing the rigidity of static workflows and token-level RL by introducing a POMDP-based autonomous agent and Proximal Sequence Policy Optimization (PSPO). It enables dynamic, context-aware decisions to invoke Search or Expand tools, with a reward design that emphasizes novel, highly relevant papers while penalizing redundant calls. Empirical results on AutoScholarQuery and RealScholarQuery show that PaperScout with PSPO achieves superior recall and semantic relevance scores compared to baselines, with faster convergence and more stable optimization. This approach offers a practical path toward flexible, scalable, and interpretably adaptive multi-turn academic search, with potential applicability beyond literature retrieval to any sequential, tool-augmented discovery task.

Abstract

Academic paper search is a fundamental task in scientific research, yet most existing approaches rely on rigid, predefined workflows that struggle with complex, conditional queries. To address this limitation, we propose PaperScout, an autonomous agent that reformulates paper search as a sequential decision-making process. Unlike static workflows, PaperScout dynamically decides whether, when, and how to invoke search and expand tools based on accumulated retrieval context. However, training such agents presents a fundamental challenge: standard reinforcement learning methods, typically designed for single-turn tasks, suffer from a granularity mismatch when applied to multi-turn agentic tasks, where token-level optimization diverges from the granularity of sequence-level interactions, leading to noisy credit assignment. We introduce Proximal Sequence Policy Optimization (PSPO), a process-aware, sequence-level policy optimization method that aligns optimization with agent-environment interaction. Comprehensive experiments on both synthetic and real-world benchmarks demonstrate that PaperScout significantly outperforms strong workflow-driven and RL baselines in both recall and relevance, validating the effectiveness of our adaptive agentic framework and optimization strategy.

Figures (6)

• Figure 1: Comparison of different academic search paradigms: Semantic Match, Fixed Workflow, and Agentic Search.
• Figure 2: Overview of PaperScout and the motivation for PSPO.Left: PaperScout models multi-turn paper search as a POMDP by maintaining a paper pool as the latent state; at each step, the agent observes a summarized view of the pool and issues Search or Expand tool calls to retrieve new papers, which are then merged to update the pool. Right:Granularity mismatch in credit assignment: PPO attributes a single step reward to many tokens in the response, leading to diluted learning signals, whereas PSPO treats each complete response as the atomic action and performs advantage estimation and policy updates at the sequence level.
• Figure 3: Average recall as tool calls accumulate on RealScholarQuery (top) and AutoScholarQuery (bottom). PSPO consistently yields higher recall with the same number of tool calls.
• Figure 4: Training dynamics and optimization statistics for different policy optimization methods. (a) Trajectory returns during training. PSPO converges faster and reaches a higher return plateau than PPO and GSPO. (b) Actor gradient norm during policy updates. PSPO maintains a smaller gradient norm with a clearer downward trend than PPO. (c) Critic loss during training. PSPO is consistently lower than PPO, especially in early training.
• Figure 5: Search–Expand call distribution across queries for different models. The trained PaperScout (Qwen3-4B-PSPO) exhibits a broader and more balanced allocation of Search and Expand calls compared with the untrained Qwen3-4B and Qwen3-Max, indicating more flexible multi-turn retrieval behavior.