Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspective

TL;DR

This paper tackles inconsistent claims in reinforcement fine-tuning of LLMs by disentangling design choices through a bottom-up experimental pipeline that maps RLFT to a multi-armed bandit framework with a huge action space and deterministic rewards. Starting from a minimal setup of one training example and one rollout per round, the authors progressively add components (advantage design, rollout count, data difficulty, reward design, base model) to study their effects on training dynamics and generalization across three models and two reasoning datasets. Key findings show that, in moderate settings, the optimal policy can be learned without relying on advantage design; increasing rollouts speeds learning but does not reliably improve generalization; extreme data conditions or negative rewards can drastically degrade performance, with base-model differences like OLMo diverging in generalization. The work provides a practical pipeline to diagnose confounds, identifies bottlenecks in generalization under challenging data, and offers guidance to prioritize robustness over scaling in reinforcement fine-tuning.

Abstract

A large number of heuristics have been proposed to optimize the reinforcement fine-tuning of LLMs. However, inconsistent claims are made from time to time, making this area elusive. Reflecting on this situation, two fundamental questions still lack a clear understanding: 1) what is the role of each optimizing choice? 2) which ones are the bottlenecks? This paper aims to shed light on them, and it faces the challenge of several entangled confounding factors in the fine-tuning process. To tackle this challenge, we propose a bottom-up experiment pipeline. The bottom layer is composed of a minimalist configuration: one training data, one rollout per round and the reward directly serve as the learning signal without advantage function design. This minimalist configuration connects to multi-armed bandit learning with extremely large discrete action space, which offers theories to corroborate the experiment findings. The up procedure of the experiment pipeline expanding the minimalist configuration layer by layer, examining the role of each design choice. Experimental results on three LLMs and two reasoning datasets not only reveal new understanding of the design choice but also yield essential insights to shape the area.

Figures (8)

• Figure 1: LLaMA fine-tuning under minimalist configuration.
• Figure 2: Qwen fine-tuning under minimalist configuration.
• Figure 3: Impact of advantage function (Math dataset).
• Figure 4: Impact of number of rollouts (Math dataset).
• Figure 5: Impact of extremely difficult data (without advantage).