Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs

Bowen Sun, Yujun Cai, Ming-Hsuan Yang, Hang Wu, Yiwei Wang

TL;DR

Video LLMs exhibit temporal instability when RoPE-based encodings extend to time via M-RoPE, causing attention to swing with small frame-rate changes. The authors propose Phase Aggregated Smoothing (PAS), a training-free inference plug-in that assigns small opposed phase shifts across attention heads and aggregates their outputs, effectively smoothing the time-domain modulation $m(\Delta)$ while preserving per-head spectra. They prove that smoother $m(\Delta)$ yields Lipschitz-stable logits, that multi-phase averaging attenuates high-frequency ripples, and that per-head spectra remain invariant under Nyquist sampling, so PAS changes sampling/aggregation rather than encoding. Empirically, PAS yields consistent gains across nine benchmarks with negligible overhead, especially under sparse temporal sampling, and provides practical defaults (e.g., two head groups with offsets [0, 0.5]). Overall, PAS offers a simple, robust, drop-in enhancement for phase-robust temporal encoding in Video LLMs with broad applicability to low-FPS regimes and frame-merged inputs.

Abstract

Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames. We trace this instability to the common extension of Rotary Position Embeddings to video through multimodal RoPE. The induced inverse Fourier time kernel exhibits frame-scale ripples that multiply adjacent frames by different factors, which perturbs attention that should otherwise be governed by the raw query key inner product. We present Phase Aggregated Smoothing (PAS), a simple, training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs. PAS preserves the per-head spectrum magnitude, while the aggregation effectively smooths the temporal kernel and reduces phase sensitivity without changing the positional encoding structure. Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling. Experiments on multiple video understanding benchmarks under matched token budgets show consistent improvements with negligible computational overhead. PAS provides a plug and play upgrade for robust temporal encoding in Video LLMs.

PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs

TL;DR

Video LLMs exhibit temporal instability when RoPE-based encodings extend to time via M-RoPE, causing attention to swing with small frame-rate changes. The authors propose Phase Aggregated Smoothing (PAS), a training-free inference plug-in that assigns small opposed phase shifts across attention heads and aggregates their outputs, effectively smoothing the time-domain modulation while preserving per-head spectra. They prove that smoother yields Lipschitz-stable logits, that multi-phase averaging attenuates high-frequency ripples, and that per-head spectra remain invariant under Nyquist sampling, so PAS changes sampling/aggregation rather than encoding. Empirically, PAS yields consistent gains across nine benchmarks with negligible overhead, especially under sparse temporal sampling, and provides practical defaults (e.g., two head groups with offsets [0, 0.5]). Overall, PAS offers a simple, robust, drop-in enhancement for phase-robust temporal encoding in Video LLMs with broad applicability to low-FPS regimes and frame-merged inputs.

Abstract

Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames. We trace this instability to the common extension of Rotary Position Embeddings to video through multimodal RoPE. The induced inverse Fourier time kernel exhibits frame-scale ripples that multiply adjacent frames by different factors, which perturbs attention that should otherwise be governed by the raw query key inner product. We present Phase Aggregated Smoothing (PAS), a simple, training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs. PAS preserves the per-head spectrum magnitude, while the aggregation effectively smooths the temporal kernel and reduces phase sensitivity without changing the positional encoding structure. Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling. Experiments on multiple video understanding benchmarks under matched token budgets show consistent improvements with negligible computational overhead. PAS provides a plug and play upgrade for robust temporal encoding in Video LLMs.

Paper Structure

This paper contains 29 sections, 4 theorems, 35 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Preliminaries
  6. Theory
  7. Implementation
  8. Experiment
  9. Datasets and Evaluation
  10. Main Results
  11. Parameter Insensitivity of PAS
  12. Ablation: Impact of Sampling Rate on PAS
  13. Conclusion
  14. Setup.
  15. Assumptions.
  16. ...and 14 more sections

Key Result

Theorem 1

Let $\Omega=\{\omega_i\}_{i=0}^{m-1}$ be the RoPE lines and define the IFT kernel $m(\Delta)\;:=\;\frac{1}{m}\sum_{i=0}^{m-1} e^{\mathrm{j}\omega_i \Delta}.$ Assume (i) a large number of lines $m$, (ii) near-uniform spectral energy across lines (the content-dependent coefficients $\{C_i\}$ have simi

Figures (5)

  • Figure 1: A failure from a real clip yt:CJHP6dPjuGY. Temporal RoPE imposes an interval dependent gain on attention; when a key frame lies in a low gain trough of the modulation, it is down weighted compared to less relevant frames, which propagates to downstream errors.
  • Figure 2: Temporal non-smoothness of the time domain modulation from original M-RoPE (upper) and how Phase Aggregated Smoothing (PAS) mitigates it (lower). PAS assigns small, opposed phase shifts to the query stream per head. Each head preserves its spectrum magnitude because a time shift only rotates phases. Head aggregation then acts as a controlled moving average in time, producing a smoother effective modulation across adjacent frames and reducing low gain induced suppression of key frames.
  • Figure 3: Implementation of PAS. For the $Q$ matrix of each vision token, we apply temporal offsets to each query head.
  • Figure 4: Offset sweep with fixed $K{=}2$. Offsets $\Delta$ are applied to only one of the two groups of query heads in bin units. We report accuracy as a function of $\Delta$.
  • Figure 5: Sampling ratio ablation with fixed $K{=}2$, $\Delta{=}0.5$. Classification accuracy as a function of the sampling ratio $r$.

Theorems & Definitions (4)

  • Theorem 1: Phase modulation approximation
  • Theorem 2: Smooth IFT $\Rightarrow$ Phase-Stable Attention
  • Theorem 3: Multi-Phase Averaging Smooths the IFT Waveform
  • Theorem 4: Per-Head Recovered Spectrum is Invariant under Temporal Offsets