Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement

Yunjian Zhang, Sudong Wang, Yang Li, Peiran Xu, Conghao Zhou, Xiaoyue Ma, Jianing Li, Yao Zhu

TL;DR

This work tackles the high data and compute demands of reinforcement learning with verifiable rewards (RLVR) for reasoning in large language models. It establishes a theoretical lower bound showing strong reasoning can be activated with a minimalist data regime and introduces Dynamic One-Shot Policy Refinement (DoPR), a lightweight, uncertainty-aware strategy that selects a single most informative training sample per batch and concentrates rollouts on it using an EM-UCB acquisition score. Empirically, DoPR achieves competitive reasoning accuracy on diverse mathematical benchmarks while reducing rollout costs by nearly an order of magnitude, and it remains robust across data-scarce settings. Overall, the paper provides both theoretical and algorithmic contributions toward practical, resource-efficient RL-based reasoning for LLMs, with significant implications for scalable deployment in resource-limited environments.

Abstract

Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), an uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.

Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement

TL;DR

This work tackles the high data and compute demands of reinforcement learning with verifiable rewards (RLVR) for reasoning in large language models. It establishes a theoretical lower bound showing strong reasoning can be activated with a minimalist data regime and introduces Dynamic One-Shot Policy Refinement (DoPR), a lightweight, uncertainty-aware strategy that selects a single most informative training sample per batch and concentrates rollouts on it using an EM-UCB acquisition score. Empirically, DoPR achieves competitive reasoning accuracy on diverse mathematical benchmarks while reducing rollout costs by nearly an order of magnitude, and it remains robust across data-scarce settings. Overall, the paper provides both theoretical and algorithmic contributions toward practical, resource-efficient RL-based reasoning for LLMs, with significant implications for scalable deployment in resource-limited environments.

Abstract

Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), an uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.
Paper Structure (15 sections, 2 theorems, 38 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 2 theorems, 38 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Large Language Models
  4. Reinforcement Learning with Verifiable Rewards
  5. Rethinking Data Requirements for RLVR
  6. Dynamic One-Shot Policy Refinement
  7. Experiments
  8. Experimental Settings
  9. Benchmarking Across Diverse Tasks
  10. Impact of Training Data Scale
  11. Comparison under Equal Rollout Budgets
  12. Ablation Studies
  13. Efficiency in Rollout and Update Time
  14. Conclusion
  15. Detailed Proof of Theorem 1

Key Result

Theorem 3.1

Consider an optimization procedure where the policy is updated using a single sample at each step, let $\pi_{\theta^\star}$ denotes the optimal policy and $\pi_{\theta_N}$ denotes the policy after $N$ updates. To guarantee that the expected performance gap satisfies it suffices that the number of steps $N$ satisfies where $\epsilon$ denotes the initial performance gap.

Figures (3)

  • Figure 1: Accuracy on MATH and Minierva-MATH with varying training set sizes. Except for the 8-sample setting, all configurations converge to comparable performance, indicating that strong reasoning ability can be achieved with few training examples.
  • Figure 2: Overview of DoPR, which dynamically selects a single high-value training instance from each mini-batch based on historical reward statistics.
  • Figure 3: DoPR consistently yields shorter reasoning trajectories and faster update cycles, indicating improved runtime efficiency and a reduced computational burden for training.

Theorems & Definitions (4)

  • Theorem 3.1
  • proof
  • Theorem A.1
  • proof