PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient
Zijian Wang, Tiancheng Huang, Hanqi Li, Da Ma, Lu Chen, Kai Yu
TL;DR
PaperGuide addresses the inefficiency of small LLMs in long-horizon Paper-QA by separating planning from execution through a Draft-and-Follow framework and optimizing both stages with Draft-and-Follow Policy Optimization (DFPO). It introduces DTFT for targeted pretraining of drafts and tool-use behavior, and a reward-router with negative sample masking to stabilize learning and credit assignment. Theoretical analysis shows gradient decomposition into a standard M-GRPO term plus a draft-induced bias, supporting stable training, while empirical results on AirQA-Real and SciDQA demonstrate notable efficiency gains (I-Avg) and competitive accuracy with 3B and 7B models, approaching larger baselines. Overall, PaperGuide demonstrates that hierarchical planning and specialized training can substantially improve the efficiency and reliability of small LLM agents in scientific QA tasks, with practical implications for resource-constrained deployments.
Abstract
The accelerating growth of the scientific literature makes it increasingly difficult for researchers to track new advances through manual reading alone. Recent progress in large language models (LLMs) has therefore spurred interest in autonomous agents that can read scientific papers and extract task-relevant information. However, most existing approaches rely either on heavily engineered prompting or on a conventional SFT-RL training pipeline, both of which often lead to excessive and low-yield exploration. Drawing inspiration from cognitive science, we propose PaperCompass, a framework that mitigates these issues by separating high-level planning from fine-grained execution. PaperCompass first drafts an explicit plan that outlines the intended sequence of actions, and then performs detailed reasoning to instantiate each step by selecting the parameters for the corresponding function calls. To train such behavior, we introduce Draft-and-Follow Policy Optimization (DFPO), a tailored RL method that jointly optimizes both the draft plan and the final solution. DFPO can be viewed as a lightweight form of hierarchical reinforcement learning, aimed at narrowing the `knowing-doing' gap in LLMs. We provide a theoretical analysis that establishes DFPO's favorable optimization properties, supporting a stable and reliable training process. Experiments on paper-based question answering (Paper-QA) benchmarks show that PaperCompass improves efficiency over strong baselines without sacrificing performance, achieving results comparable to much larger models.