HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization

TL;DR

HAPO addresses the inefficiency of verbose reasoning in LLMs by introducing a history-aware length reward that leverages a per-query history state $h_i$ to bias toward shorter correct solutions. By combining an accuracy reward with a length reward $rl_i^{(j)}$, and updating $h_i$ across training encounters, HAPO encourages iterative compression of correct solutions without over-penalizing exploratory attempts. Empirical results on GSM8K, MATH500, and AIME2024 show substantial length reductions (33–59%) with modest accuracy losses (2–5%), outperforming universal-budget and query-level baselines in overall length-correctness trade-offs. The approach demonstrates strong potential for reducing token usage and inference costs in reasoning tasks, while preserving practical performance gains and offering a path toward more efficient, history-informed training of LLMs.

Abstract

While scaling the length of responses at test-time has been shown to markedly improve the reasoning abilities and performance of large language models (LLMs), it often results in verbose outputs and increases inference cost. Prior approaches for efficient test-time scaling, typically using universal budget constraints or query-level length optimization, do not leverage historical information from previous encounters with the same problem during training. We hypothesize that this limits their ability to progressively make solutions more concise over time. To address this, we present History-Aware Policy Optimization (HAPO), which keeps track of a history state (e.g., the minimum length over previously generated correct responses) for each problem. HAPO employs a novel length reward function based on this history state to incentivize the discovery of correct solutions that are more concise than those previously found. Crucially, this reward structure avoids overly penalizing shorter incorrect responses with the goal of facilitating exploration towards more efficient solutions. By combining this length reward with a correctness reward, HAPO jointly optimizes for correctness and efficiency. We use HAPO to train DeepSeek-R1-Distill-Qwen-1.5B, DeepScaleR-1.5B-Preview, and Qwen-2.5-1.5B-Instruct, and evaluate HAPO on several math benchmarks that span various difficulty levels. Experiment results demonstrate that HAPO effectively induces LLMs' concise reasoning abilities, producing length reductions of 33-59% with accuracy drops of only 2-5%.

Figures (5)

• Figure 1: Results averaged across GSM8K, MATH500, and AIME2024. Pass@1 is the average accuracy across multiple sampled responses per query; #Tokens is the average number of tokens in the responses. Compared with the base model DeepSeek-R1-1.5B, HAPO significantly reduces response length while preserving accuracy, giving a better tradeoff than other baselines.
• Figure 2: Left: Illustration of the HAPO reward computation and history state ($h_i$) update over three encounters with problem $x_i$, assuming one response is sampled per encounter ($|Y_{i}^{(j)}|=1$). Initially ($j=1$, $h_i=\texttt{Null}$). The first correct response $y_{i}^{(1)}$ (length 500) receives a zero length reward ($rl=0$) and an overall reward of 1 ($r=1$), and updates $h_i$ to 500. In the second encounter ($j=2$), an incorrect response $y_{i}^{(2)}$ (length 400) receives $rl=0$ and $r=0$, and does not update $h_i$. In the third encounter ($j=3$), a correct response $y_{i}^{(3)}$ (length 167), being shorter than the current $h_i=500$, receives a positive length reward ($rl=0.87$) and overall reward ($r \ge 1$), and updates $h_i$ to 167. Right: The HAPO length reward function $rl_i^{(j)}$ plotted against response length $|y_i^{(j)}|$. Green curve: reward for correct responses ($a_{i}^{(j)}=a_i^*$), incentivizing lengths shorter than $h_i$; Red curve: reward for incorrect responses ($a_{i}^{(j)} \neq a_i^*$). Both curves are centered at $(h_i, 0)$. The initial case where $h_i$ is Null is omitted for visual clarity.
• Figure 3: Comparison between HAPO and DeepSeek-R1-1.5B in terms of Pass@1 and #Tokens across questions of varying difficulty levels.
• Figure 4: Trajectories of average response length $|y_i|$ and $h_i$ on the training set (left) and validation set (right), computed at the end of each epoch. For the training set, the value of $|y_i|$ at each epoch is obtained by averaging the lengths of responses sampled from $p_\theta$ during that epoch across the entire training set. Similarly, $h_i$ is averaged over all training examples, excluding Null values. Note that $h_i$ is measured after the update; that is, in epoch $j$, $h_i = \mathrm{aggre}({L_i^{(1)}, \cdots, L_i^{(j)}})$, which includes the latest length $L_i^{(j)}$.
• Figure 5: Visualization of results on GSM8K, MATH500, and AIME2024, with DeepSeek-R1-1.5B as the base model.