Online Causal Kalman Filtering for Stable and Effective Policy Optimization

TL;DR

Online Causal Kalman Filtering for stable and effective Policy Optimization (KPO) is proposed, which model the desired IS ratio as a latent state that evolves across tokens and applies a Kalman filter to update this state online and autoregressively based on the states of past tokens, regardless of future tokens.

Abstract

Reinforcement learning for large language models suffers from high-variance token-level importance sampling (IS) ratios, which would destabilize policy optimization at scale. To improve stability, recent methods typically use a fixed sequence-level IS ratio for all tokens in a sequence or adjust each token's IS ratio separately, thereby neglecting temporal off-policy derivation across tokens in a sequence. In this paper, we first empirically identify that local off-policy deviation is structurally inconsistent at the token level, which may distort policy-gradient updates across adjacent tokens and lead to training collapse. To address the issue, we propose Online Causal Kalman Filtering for stable and effective Policy Optimization (KPO). Concretely, we model the desired IS ratio as a latent state that evolves across tokens and apply a Kalman filter to update this state online and autoregressively based on the states of past tokens, regardless of future tokens. The resulting filtered IS ratios preserve token-wise local structure-aware variation while strongly smoothing noise spikes, yielding more stable and effective policy updates. Experimentally, KPO achieves superior results on challenging math reasoning datasets compared with state-of-the-art counterparts.

Figures (4)

• Figure 1: Illustration of local structural off-policy patterns. Raw token-level importance-sampling (IS) ratios (blue) exhibit high local variance and structural inconsistency, whereas a sequence-level IS ratio (purple) is globally smooth but obscures within-sequence structure. Off-policy frequency increases over the sequence (window-wise statistics), off-policy runs are short-lived (run-length distribution), and token states switch frequently, suggesting weak local coherence. Token-level Kalman filtering (red) yields locally smoothed yet structurally consistent IS ratios.
• Figure 2: Causal Kalman filtering. The filter alternates prediction, adaptive gain computation, and update to produce a smoothed estimate $\hat{\rho}_{t|t}$ and its uncertainty $P_{t|t}$ from streaming observations $z_t$, using process noise $Q$ and observation noise $V$.
• Figure 3: Training dynamics of KPO-clipped over optimization steps. From left to right: mean episodic reward, policy entropy, PPO clip fraction, and policy gradient loss. Solid lines denote the average across runs, and the shaded bands indicate variability across runs.
• Figure 4: Effect of the Kalman filter noise ratio $Q/V$ on training dynamics. From left to right, we report the mean episodic reward and the PPO clip fraction. Solid lines show the mean over multiple runs, and shaded regions indicate standard deviation.