Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF

Ziyuan Gao, Di Liang, Xianjie Wu, Philippe Morel, Minlong Peng

TL;DR

DeCoRL addresses fundamental inefficiencies and opacity in traditional RLHF chain-of-thought reasoning by decoupling reasoning into parallel, specialized sub-steps managed by a modular ensemble. It couples parallel sub-step generation with a dual-reward attribution mechanism and cascaded reinforcement (DRPO) to coordinate module objectives while preserving inter-step dependencies. The approach achieves substantial speedups ($$3.8\times$$ latency), dramatic energy savings ($$72.4\%$$), and improved interpretability ($$22.7\%$$ error localization) without sacrificing solution quality across RM-Bench, RMB, and RewardBench. Ablation studies confirm the critical role of contribution rewards and cascaded training in robust error diagnosis and performance. The framework supports dynamic expansion and hardware-aware deployment, enabling real-time, interpretable reasoning in complex tasks.

Abstract

Existing reinforcement learning methods for Chain-of-Thought reasoning suffer from two critical limitations. First, they operate as monolithic black boxes that provide undifferentiated reward signals, obscuring individual step contributions and hindering error diagnosis. Second, sequential decoding has O(n) time complexity. This makes real-time deployment impractical for complex reasoning tasks. We present DeCoRL (Decoupled Reasoning Chains via Coordinated Reinforcement Learning), a novel framework that transforms reasoning from sequential processing into collaborative modular orchestration. DeCoRL trains lightweight specialized models to generate reasoning sub-steps concurrently, eliminating sequential bottlenecks through parallel processing. To enable precise error attribution, the framework designs modular reward functions that score each sub-step independently. Cascaded DRPO optimization then coordinates these rewards while preserving inter-step dependencies. Comprehensive evaluation demonstrates state-of-the-art results across RM-Bench, RMB, and RewardBench, outperforming existing methods including large-scale models. DeCoRL delivers 3.8 times faster inference while maintaining superior solution quality and offers a 22.7\% improvement in interpretability through explicit reward attribution. These advancements, combined with a 72.4\% reduction in energy consumption and a 68\% increase in throughput, make real-time deployment of complex reasoning systems a reality.

DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF

TL;DR

DeCoRL addresses fundamental inefficiencies and opacity in traditional RLHF chain-of-thought reasoning by decoupling reasoning into parallel, specialized sub-steps managed by a modular ensemble. It couples parallel sub-step generation with a dual-reward attribution mechanism and cascaded reinforcement (DRPO) to coordinate module objectives while preserving inter-step dependencies. The approach achieves substantial speedups ( latency), dramatic energy savings (), and improved interpretability ( error localization) without sacrificing solution quality across RM-Bench, RMB, and RewardBench. Ablation studies confirm the critical role of contribution rewards and cascaded training in robust error diagnosis and performance. The framework supports dynamic expansion and hardware-aware deployment, enabling real-time, interpretable reasoning in complex tasks.

Abstract

Existing reinforcement learning methods for Chain-of-Thought reasoning suffer from two critical limitations. First, they operate as monolithic black boxes that provide undifferentiated reward signals, obscuring individual step contributions and hindering error diagnosis. Second, sequential decoding has O(n) time complexity. This makes real-time deployment impractical for complex reasoning tasks. We present DeCoRL (Decoupled Reasoning Chains via Coordinated Reinforcement Learning), a novel framework that transforms reasoning from sequential processing into collaborative modular orchestration. DeCoRL trains lightweight specialized models to generate reasoning sub-steps concurrently, eliminating sequential bottlenecks through parallel processing. To enable precise error attribution, the framework designs modular reward functions that score each sub-step independently. Cascaded DRPO optimization then coordinates these rewards while preserving inter-step dependencies. Comprehensive evaluation demonstrates state-of-the-art results across RM-Bench, RMB, and RewardBench, outperforming existing methods including large-scale models. DeCoRL delivers 3.8 times faster inference while maintaining superior solution quality and offers a 22.7\% improvement in interpretability through explicit reward attribution. These advancements, combined with a 72.4\% reduction in energy consumption and a 68\% increase in throughput, make real-time deployment of complex reasoning systems a reality.

Paper Structure

This paper contains 24 sections, 12 equations, 2 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain-of-Thought Reasoning and Decomposition
  4. Reinforcement Learning for Reasoning
  5. Parallel Processing and Modular Architectures
  6. Process Supervision and Step-Level Feedback
  7. Methodology
  8. Parallel Generation Architecture
  9. Module Specialization
  10. Interface Standardization
  11. Contextual Isolation
  12. Dual-Reward Attribution Mechanism
  13. Local Reward
  14. Contribution Reward
  15. Integrated Reward
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Sequential Approach vs. DeCoRL Framework: Solo pianist represents monolithic sequential reasoning with limited capacity. Symphony orchestra illustrates our collaborative modular approach with specialized sub-models working in parallel coordination under unified guidance.
  • Figure 2: DeCoRL Framework Architecture: The complete pipeline from problem decomposition through sub-step generation, parallel model training, reward model evaluation, and Cascaded DRPO optimization for modular ensemble coordination.