Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agents

TL;DR

DAVID-GRPO tackles the challenge of multi-hop reasoning with resource-constrained, small-language-model agents. It introduces three synergistic components—a few-shot warm-start, grounded retrieval rewards, and grounded expansion—that provide dense, evidence-grounded feedback and targeted exploration under strict memory budgets. Empirical results on six multi-hop QA benchmarks with models up to 1.5B parameters and training on four RTX 3090 GPUs show state-of-the-art performance within low-budget regimes, often surpassing high-budget baselines. The work demonstrates that with the right inductive biases and reward design, compact models can achieve robust, evidence-grounded multi-hop reasoning at a fraction of the computational cost, broadening access to advanced RL-based reasoning. These findings highlight the importance of process-level feedback and adaptive expansion in democratizing high-level reasoning capabilities.

Abstract

While reinforcement learning (RL) has empowered multi-turn reasoning agents with retrieval and tools, existing successes largely depend on extensive on-policy rollouts in high-cost, high-accuracy regimes. Under realistic resource constraints that cannot support large models or dense explorations, however, small language model agents fall into a low-cost, low-accuracy regime, where limited rollout budgets lead to sparse exploration, sparse credit assignment, and unstable training. In this work, we challenge this trade-off and show that small language models can achieve strong multi-hop reasoning under resource constraints. We introduce DAVID-GRPO, a budget-efficient RL framework that (i) stabilizes early learning with minimal supervision, (ii) assigns retrieval credit based on evidence recall, and (iii) improves exploration by resampling truncated near-miss trajectories. Evaluated on agents up to 1.5B parameters trained on only four RTX 3090 GPUs, DAVID-GRPO consistently outperforms prior RL methods designed for large-scale settings on six multi-hop QA benchmarks. These results show that with the right inductive biases, small agents can achieve low training cost with high accuracy.

Figures (6)

• Figure 1: Average exact match (EM) across four multi-hop QA benchmarks versus rollouts per batch (log scale) using Qwen2.5-1.5B. Shading indicates rollout intensity. The dashed line illustrates the scaling trend for Tree-GRPO. In the low-cost regime, David-GRPO outperforms StepSearch, Search-R1-v0.3, and Tree-GRPO, achieving parity with Tree-GRPO's high-cost performance while using only 4.7% of its budget.
• Figure 2: Overview of David-GRPO.
• Figure 3: EM and average number of unique retrieval actions on HotpotQA by user question types with Qwen2.5-1.5B. Search-R1-v0.3 is trained along with its retrieval reward.
• Figure 4: EM and average number of unique retrieval actions on MuSiQue by reasoning hops with Qwen2.5-1.5B. Dashed lines indicate the minimum number of retrieval actions required for each hop subset. Search-R1-v0.3 is trained along with its retrieval reward.
• Figure 5: Analysis on Warmup Strategies. Performance comparison of different warmup methods applied before the GRPO phase with grounded retrieval reward on Qwen2.5-1.5B.