TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation

TL;DR

TokenTrim addresses temporal drift in autoregressive long video generation by performing inference-time latent-token pruning. It detects unstable tokens through per-token drift between consecutive latent summaries and prunes them from the KV cache, optionally regenerating the current batch; motion-stabilized initialization further aids early coherence. The approach requires no training or architectural changes and yields consistent improvements in semantic fidelity and visual quality across two inference strategies, with modest computational overhead and strong human preference. These findings highlight the value of selective latent-space forgetting as a practical path to durable long-horizon video synthesis.

Abstract

Auto-regressive video generation enables long video synthesis by iteratively conditioning each new batch of frames on previously generated content. However, recent work has shown that such pipelines suffer from severe temporal drift, where errors accumulate and amplify over long horizons. We hypothesize that this drift does not primarily stem from insufficient model capacity, but rather from inference-time error propagation. Specifically, we contend that drift arises from the uncontrolled reuse of corrupted latent conditioning tokens during auto-regressive inference. To correct this accumulation of errors, we propose a simple, inference-time method that mitigates temporal drift by identifying and removing unstable latent tokens before they are reused for conditioning. For this purpose, we define unstable tokens as latent tokens whose representations deviate significantly from those of the previously generated batch, indicating potential corruption or semantic drift. By explicitly removing corrupted latent tokens from the auto-regressive context, rather than modifying entire spatial regions or model parameters, our method prevents unreliable latent information from influencing future generation steps. As a result, it significantly improves long-horizon temporal consistency without modifying the model architecture, training procedure, or leaving latent space.

Figures (11)

• Figure 1: Text-to-video results before and after applying TokenTrim on Rolling Forcing liu2025rolling and Self Forcing huang2025self.
• Figure 2: TokenTrim overview at autoregressive step $t$. (a) Given the candidate batch $\mathcal{X}_t$ and the previous batch $\mathcal{X}_{t-1}$, we encode each frame and form latent summaries $Z_t$ and $Z_{t-1}$ by averaging latents over the $F$ frames in each batch. We compute per-token drift $d_i=\lVert Z_t(i)-Z_{t-1}(i)\rVert_2$ and select the top-$pN$ largest drifts to form the unstable set $S_t$, from which we compute the drift severity $D_t$. (b) We compare $D_t$ to the adaptive threshold $\mu_t+\lambda\sigma_t$. If $D_t \le \mu_t+\lambda\sigma_t$, the KV cache $(K,V)$ is left unchanged and the batch is accepted. Otherwise, we mask the selected token positions in the temporal KV cache to obtain $(\tilde{K},\tilde{V})$ and regenerate the current batch conditioned on the pruned cache. Running statistics and the cache are updated using the accepted batch $\mathcal{X}_t^{\star}$.
• Figure 3: Qualitative results. Text-to-video results before and after applying TokenTrim on Rolling Forcing liu2025rolling and Self Forcing huang2025self. TokenTrim mitigates degradation over time, e.g., color shifts (c - background and girl, d - background and bear), artifacts (b - light lens flare) and unnatural motion (a - Pikatchu). For additional qualitative results see App. \ref{['app:additional_qualitative_self']} and App. \ref{['app:additional_qualitative_rolling']}.
• Figure 4: Text-to-video results from FlowMo shaulov2025flowmo and TokenTrim. For additional qualitative results see App. \ref{['app:additional_qualitative_flowmo']}.
• Figure 5: Human preference study conducted on Rolling Forcing liu2025rolling (left) and Self Forcing huang2025self (right) using VideoJAM-bench chefer2025videojam. TokenTrim is consistently preferred in terms of drift reduction, motion quality, and overall visual quality, while preserving text–video alignment. Error bars indicate 95% confidence intervals computed via Dirichlet sampling with Laplace smoothing.