Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning

TL;DR

This work reframes temperature control in LLM reinforcement learning as a learnable meta-policy, addressing the rigidity of fixed or heuristic temperature schedules. TAMPO introduces a hierarchical two-loop framework: an inner loop optimizes the LLM policy at temperatures sampled from a learnable meta-policy, while an outer loop updates the temperature distribution based on trajectory-specific advantages derived from temperature-wise likelihoods, reusing existing rollouts to avoid extra data collection. By normalizing trajectory likelihoods across candidate temperatures and weighting trajectory advantages accordingly, TAMPO derives a temperature-specific advantage that guides the meta-policy toward beneficial temperatures with EMA-based stability. Across five mathematical reasoning benchmarks, TAMPO yields consistent improvements over fixed-temperature baselines, demonstrating effective online exploration–exploitation adaptation and potential applicability to broader LLM RL settings.

Abstract

Temperature is a crucial hyperparameter in large language models (LLMs), controlling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence. Yet static or heuristic temperature schedules fail to adapt to the dynamic demands of reinforcement learning (RL) throughout training, often limiting policy improvement. We propose Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control as a learnable meta-policy. TAMPO operates through a hierarchical two-loop process. In the inner loop, the LLM policy is updated (e.g., using GRPO) with trajectories sampled at the temperature selected by the meta-policy. In the outer loop, meta-policy updates the distribution over candidate temperatures by rewarding those that maximize the likelihood of high-advantage trajectories. This trajectory-guided, reward-driven mechanism enables online adaptation without additional rollouts, directly aligning exploration with policy improvement. On five mathematical reasoning benchmarks, TAMPO outperforms baselines using fixed or heuristic temperatures, establishing temperature as an effective learnable meta-policy for adaptive exploration in LLM reinforcement learning. Accepted at ICLR 2026.

Figures (5)

• Figure 1: Overview of Temperature Adaptive Meta Policy Optimization (TAMPO). The framework operates through a hierarchical two-loop process. In the inner loop, the LLM policy is optimized with critic-free RL (e.g., GRPO) using rollouts sampled at the temperature chosen by the meta-policy. In the outer loop, the meta-policy is updated by evaluating trajectory likelihoods under virtual temperatures, deriving temperature-specific advantages ($\mathcal{A}_{i}^{(T_k)} = \hat{\ell}_{T_k}(\tau_i) \cdot A_i$ for trajectory $\tau_i$ w.r.t. virtual temperature $T_k$), and reinforcing those that yield high-advantage rollouts (see §\ref{['sec:approach']}). This design establishes temperature as a learnable meta-policy, enabling online adaptation and effective optimization of LLM policy without extra rollouts.
• Figure 2: Example of trajectory likelihood under different temperatures for 8 rollouts of a prompt.
• Figure 3: Sampling temperatures from the meta-policy using nucleus sampling.
• Figure 4: Distribution of trajectory likelihood-optimal temperatures under three fixed training temperatures, respectively. Green curve corresponds to the likelihood-optimal temperatures of positive-advantage trajectories ($A>0$), red curve to negative-advantage trajectories ($A<0$), and blue curve to the sampled fixed temperature.
• Figure 5: System prompt for the policy model.