Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Models

Yan Liu, Feng Zhang, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Han Liu, Yangdong Deng

TL;DR

CoT-Flow reframes chain-of-thought reasoning as a continuous probabilistic flow, introducing Probabilistic Flow Progress to quantify step-level information gain. It enables train-free greedy flow decoding and flow-based reinforcement learning with dense, verifier-free rewards, achieving a favorable balance between accuracy and inference efficiency on challenging math and general reasoning benchmarks. The approach couples principled velocity signals with a stop-gradient mechanism and a time-weighted flow objective, yielding more concise and informative reasoning trajectories and robust optimization. Empirical results show substantial improvements in difficult tasks (e.g., AIME 2024) and a Pareto-frontier advancement in efficiency, suggesting practical impact for scalable reasoning with large language models.

Abstract

High-quality chain-of-thought has demonstrated strong potential for unlocking the reasoning capabilities of large language models. However, current paradigms typically treat the reasoning process as an indivisible sequence, lacking an intrinsic mechanism to quantify step-wise information gain. This granularity gap manifests in two limitations: inference inefficiency from redundant exploration without explicit guidance, and optimization difficulty due to sparse outcome supervision or costly external verifiers. In this work, we propose CoT-Flow, a framework that reconceptualizes discrete reasoning steps as a continuous probabilistic flow, quantifying the contribution of each step toward the ground-truth answer. Built on this formulation, CoT-Flow enables two complementary methodologies: flow-guided decoding, which employs a greedy flow-based decoding strategy to extract information-efficient reasoning paths, and flow-based reinforcement learning, which constructs a verifier-free dense reward function. Experiments on challenging benchmarks demonstrate that CoT-Flow achieves a superior balance between inference efficiency and reasoning performance.

Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Models

TL;DR

CoT-Flow reframes chain-of-thought reasoning as a continuous probabilistic flow, introducing Probabilistic Flow Progress to quantify step-level information gain. It enables train-free greedy flow decoding and flow-based reinforcement learning with dense, verifier-free rewards, achieving a favorable balance between accuracy and inference efficiency on challenging math and general reasoning benchmarks. The approach couples principled velocity signals with a stop-gradient mechanism and a time-weighted flow objective, yielding more concise and informative reasoning trajectories and robust optimization. Empirical results show substantial improvements in difficult tasks (e.g., AIME 2024) and a Pareto-frontier advancement in efficiency, suggesting practical impact for scalable reasoning with large language models.

Abstract

High-quality chain-of-thought has demonstrated strong potential for unlocking the reasoning capabilities of large language models. However, current paradigms typically treat the reasoning process as an indivisible sequence, lacking an intrinsic mechanism to quantify step-wise information gain. This granularity gap manifests in two limitations: inference inefficiency from redundant exploration without explicit guidance, and optimization difficulty due to sparse outcome supervision or costly external verifiers. In this work, we propose CoT-Flow, a framework that reconceptualizes discrete reasoning steps as a continuous probabilistic flow, quantifying the contribution of each step toward the ground-truth answer. Built on this formulation, CoT-Flow enables two complementary methodologies: flow-guided decoding, which employs a greedy flow-based decoding strategy to extract information-efficient reasoning paths, and flow-based reinforcement learning, which constructs a verifier-free dense reward function. Experiments on challenging benchmarks demonstrate that CoT-Flow achieves a superior balance between inference efficiency and reasoning performance.
Paper Structure (40 sections, 22 equations, 12 figures, 3 tables)

This paper contains 40 sections, 22 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning for Reasoning
  4. Rectified Flow and Optimal Transport
  5. Methodology
  6. Train-Free Greedy Flow Decoding
  7. Flow-Based Dense Rewards in RL
  8. Global Reward and Stop Gradient
  9. Orthogonal Decomposition of Gradients
  10. Flow Gradient Estimation
  11. Experiments
  12. Experimental Setup
  13. Datasets and Models.
  14. Baselines.
  15. Implementation Details
  16. ...and 25 more sections

Figures (12)

  • Figure 1: Illustration of our proposed CoT-Flow. Panel A: Standard CoT reasoning often exhibits a random walk behavior in the information space, characterized by unstructured exploration, redundancy, and reliance on sparse outcome-based rewards. Panel B (Ours): CoT-Flow models reasoning as a probabilistic flow. By optimizing probabilistic flow progress, it rectifies the reasoning trajectory into the shortest path from the question to the answer, providing dense supervision signals without external verifiers.
  • Figure 2: Visualization of velocity $v(s_i)$ over a chain-of-thought segment. Blue denotes higher velocity scores, and red denotes lower ones.
  • Figure 3: The posterior prompt template.
  • Figure 4: Comparison of token consumption for Standard CoT and our CoT-Flow method. CoT-Flow maintains comparable verbosity while improving accuracy.
  • Figure 5: Pareto frontier analysis of reasoning efficiency. The model accuracy (%) and computational cost (average token length) across datasets are presented.
  • ...and 7 more figures