Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs

Yongji Wu, Xueshen Liu, Haizhong Zheng, Juncheng Gu, Beidi Chen, Z. Morley Mao, Arvind Krishnamurthy, Ion Stoica

TL;DR

RLBoost addresses the resource tension between rollout and training in RL for LLMs by introducing a hybrid architecture that reserves on-demand GPUs for training and offloads rollout to elastic preemptible GPUs. It combines adaptive rollout offload with a partial response seeding strategy, token-level request migration, and pull-based weight transfer to maintain high throughput despite preemptions and fragmentation. Empirical results on 8B–32B Qwen models show throughput gains of roughly 1.51x–1.97x and cost-efficiency improvements of about 28%–49% compared with using only on-demand GPUs, while preserving synchronous RL semantics. The approach is compatible with heterogeneous hardware and offers avenues for further optimization, including broadcast-weight transfer and asynchronous RL extensions, making cost-efficient RL on LLMs more practical at scale.

Abstract

Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and training stages with fundamentally different resource requirements. Rollout typically dominates overall execution time, yet scales efficiently through multiple independent instances. In contrast, training requires tightly-coupled GPUs with full-mesh communication. Existing RL frameworks fall into two categories: co-located and disaggregated architectures. Co-located ones fail to address this resource tension by forcing both stages to share the same GPUs. Disaggregated architectures, without modifications of well-established RL algorithms, suffer from resource under-utilization. Meanwhile, preemptible GPU resources, i.e., spot instances on public clouds and spare capacity in production clusters, present significant cost-saving opportunities for accelerating RL workflows, if efficiently harvested for rollout. In this paper, we present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources. Our key insight is that rollout's stateless and embarrassingly parallel nature aligns perfectly with preemptible and often fragmented resources. To efficiently utilize these resources despite frequent and unpredictable availability changes, RLBoost adopts a hybrid architecture with three key techniques: (1) adaptive rollout offload to dynamically adjust workloads on the reserved (on-demand) cluster, (2) pull-based weight transfer that quickly provisions newly available instances, and (3) token-level response collection and migration for efficient preemption handling and continuous load balancing. Extensive experiments show RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.

RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs

TL;DR

RLBoost addresses the resource tension between rollout and training in RL for LLMs by introducing a hybrid architecture that reserves on-demand GPUs for training and offloads rollout to elastic preemptible GPUs. It combines adaptive rollout offload with a partial response seeding strategy, token-level request migration, and pull-based weight transfer to maintain high throughput despite preemptions and fragmentation. Empirical results on 8B–32B Qwen models show throughput gains of roughly 1.51x–1.97x and cost-efficiency improvements of about 28%–49% compared with using only on-demand GPUs, while preserving synchronous RL semantics. The approach is compatible with heterogeneous hardware and offers avenues for further optimization, including broadcast-weight transfer and asynchronous RL extensions, making cost-efficient RL on LLMs more practical at scale.

Abstract

Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and training stages with fundamentally different resource requirements. Rollout typically dominates overall execution time, yet scales efficiently through multiple independent instances. In contrast, training requires tightly-coupled GPUs with full-mesh communication. Existing RL frameworks fall into two categories: co-located and disaggregated architectures. Co-located ones fail to address this resource tension by forcing both stages to share the same GPUs. Disaggregated architectures, without modifications of well-established RL algorithms, suffer from resource under-utilization. Meanwhile, preemptible GPU resources, i.e., spot instances on public clouds and spare capacity in production clusters, present significant cost-saving opportunities for accelerating RL workflows, if efficiently harvested for rollout. In this paper, we present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources. Our key insight is that rollout's stateless and embarrassingly parallel nature aligns perfectly with preemptible and often fragmented resources. To efficiently utilize these resources despite frequent and unpredictable availability changes, RLBoost adopts a hybrid architecture with three key techniques: (1) adaptive rollout offload to dynamically adjust workloads on the reserved (on-demand) cluster, (2) pull-based weight transfer that quickly provisions newly available instances, and (3) token-level response collection and migration for efficient preemption handling and continuous load balancing. Extensive experiments show RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.
Paper Structure (35 sections, 2 equations, 17 figures, 5 tables, 2 algorithms)

This paper contains 35 sections, 2 equations, 17 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning for LLMs
  4. RL Frameworks
  5. Overview
  6. Design
  7. Adaptive Rollout Offload
  8. Live Request Tracking and Migration
  9. Continuous Load Balancing
  10. Pull-based Weight Transfer
  11. Implementation
  12. Evaluation
  13. Setups
  14. Overall Evaluation
  15. Analysis of Cost Efficiency
  16. ...and 20 more sections

Figures (17)

  • Figure 1: Architectures for on-policy RL frameworks.
  • Figure 2: The rollout stage dominates an RL step in the co-located architecture, yet it efficiently scales with more GPU resources as each rollout instance operates independently.
  • Figure 3: System overview of RLBoost.
  • Figure 4: RLBoost minimizes training cluster idling with an adaptive partial response seeding mechanism.
  • Figure 5: RLBoost collects responses at token granularity and migrates requests upon instance preemption, incurring only the cost of an additional prefill.
  • ...and 12 more figures