HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs

TL;DR

HiPO tackles the inefficiency of universal chain-of-thought by enabling adaptive reasoning via Think-on/Think-off. It introduces a Hybrid Data Construction Pipeline and a Hybrid RL Reward System to train models to decide when to think and when to answer concisely. Across mathematics and coding benchmarks, HiPO achieves substantial token-length reductions while preserving or improving accuracy, outperforming existing adaptive reasoning methods. This principled approach advances efficient, reasoning-oriented LLM deployment in resource-sensitive settings.

Abstract

Large Language Models (LLMs) increasingly rely on Chain-of-Thought (CoT) reasoning to improve accuracy on complex tasks. However, always generating lengthy reasoning traces is inefficient, leading to excessive token usage and higher inference costs. This paper introduces the Hybrid Policy Optimization (i.e., HiPO), a framework for adaptive reasoning control that enables LLMs to selectively decide when to engage in detailed reasoning (Think-on) and when to respond directly (Think-off). Specifically, HiPO combines a hybrid data pipelineproviding paired Think-on and Think-off responseswith a hybrid reinforcement learning reward system that balances accuracy and efficiency while avoiding over-reliance on detailed reasoning. Experiments across mathematics and coding benchmarks demonstrate that HiPO can substantially reduce token length while maintaining or improving accuracy. Finally, we hope HiPO a can be a principled approach for efficient adaptive reasoning, advancing the deployment of reasoning-oriented LLMs in real-world, resource-sensitive settings.

Figures (8)

• Figure 1: Framework of the hybrid data construction pipeline.
• Figure 2: Statistics of Data Sources.
• Figure 3: Performance of different response selection strategies.
• Figure 4: Performance of different $\gamma$ values.
• Figure 5: Performance of different rollout numbers and $\omega$ values.