Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study

Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, Xiangang Li

TL;DR

This paper examines offline reinforcement learning for enhancing long-context reasoning in large language models using LD-DPO. It demonstrates that LD-DPO yields consistent improvements across math, code generation, instruction-following, and general reasoning benchmarks, with average gains around 3.3% and notable Arena-Hard gains of 10.1%. The authors analyze the length-sensitivity of DPO and show that length-desensitized LD-DPO can produce concise, semantically rich outputs while boosting performance. They discuss future directions for additional offline methods and hybrid offline-online training to reduce cost and scale reasoning abilities.

Abstract

Despite significant advances in long-context reasoning by large language models (LLMs), primarily through Online Reinforcement Learning (RL) methods, these approaches incur substantial computational costs and complexity. In contrast, simpler and more economical Offline RL methods remain underexplored. To address this gap, we investigate the effectiveness of Offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, in enhancing the reasoning capabilities of LLMs. Extensive experiments across multiple reasoning benchmarks demonstrate that these simpler Offline RL methods substantially improve model performance, achieving an average enhancement of 3.3\%, with a particularly notable increase of 10.1\% on the challenging Arena-Hard benchmark. Furthermore, we analyze DPO's sensitivity to output length, emphasizing that increasing reasoning length should align with semantic richness, as indiscriminate lengthening may adversely affect model performance. We provide comprehensive descriptions of our data processing and training methodologies, offering empirical evidence and practical insights for developing more cost-effective Offline RL approaches.

Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study

TL;DR

This paper examines offline reinforcement learning for enhancing long-context reasoning in large language models using LD-DPO. It demonstrates that LD-DPO yields consistent improvements across math, code generation, instruction-following, and general reasoning benchmarks, with average gains around 3.3% and notable Arena-Hard gains of 10.1%. The authors analyze the length-sensitivity of DPO and show that length-desensitized LD-DPO can produce concise, semantically rich outputs while boosting performance. They discuss future directions for additional offline methods and hybrid offline-online training to reduce cost and scale reasoning abilities.

Abstract

Despite significant advances in long-context reasoning by large language models (LLMs), primarily through Online Reinforcement Learning (RL) methods, these approaches incur substantial computational costs and complexity. In contrast, simpler and more economical Offline RL methods remain underexplored. To address this gap, we investigate the effectiveness of Offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, in enhancing the reasoning capabilities of LLMs. Extensive experiments across multiple reasoning benchmarks demonstrate that these simpler Offline RL methods substantially improve model performance, achieving an average enhancement of 3.3\%, with a particularly notable increase of 10.1\% on the challenging Arena-Hard benchmark. Furthermore, we analyze DPO's sensitivity to output length, emphasizing that increasing reasoning length should align with semantic richness, as indiscriminate lengthening may adversely affect model performance. We provide comprehensive descriptions of our data processing and training methodologies, offering empirical evidence and practical insights for developing more cost-effective Offline RL approaches.
Paper Structure (18 sections, 3 equations, 4 figures, 8 tables)

This paper contains 18 sections, 3 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Description
  4. acquring quries
  5. Obtaining Chosen and Rejected Answers
  6. Overview of the LD-DPO Algorithm
  7. Experiments
  8. Benchmark
  9. Experiment Setup
  10. Results and Analysis
  11. Overall Results
  12. Analysis
  13. Instability of DPO in Specific Tasks
  14. Necessity of Meaningful Response Length Increases
  15. Related works
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Benchmark performance of DeepDistill-32B and DeepDistill-32B-lddpo on AIME 2024, GPQA-Diamond, LiveCodeBench, IFEval, and Arena-Hard. The adoption of offline RL(as seen in DeepDistill-32B-lddpo) demonstrates a consistent enhancement across all evaluated metrics.
  • Figure 2: Category distribution of queries during RL training.
  • Figure 3: Distribution of character length in chosen versus rejected responses.
  • Figure 4: Left: training loss. Right: training rewards accuracy.