Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition

TL;DR

Uni-AdaFocus addresses the heavy computational burden of video recognition by exploiting spatial, temporal, and sample-wise redundancies. It advances AdaFocus by introducing end-to-end differentiable patch localization, deformable patches, and a unified framework that dynamically allocates computation to the most informative spatial regions, frames, and samples. Key contributions include interpolation-based patch selection with deep-feature guidance, deformable patches, dynamic frame sampling with Monte Carlo-end-to-end optimization, and conditional-exit for budget-aware inference. Empirically, Uni-AdaFocus achieves state-of-the-art efficiency-accuracy trade-offs across seven benchmarks and three real-world scenarios, while remaining compatible with efficient backbones like TSM and X3D. The work points to strong practical impact for real-time video understanding and suggests future extensions to multimodal large-language models.)

Abstract

This paper presents a comprehensive exploration of the phenomenon of data redundancy in video understanding, with the aim to improve computational efficiency. Our investigation commences with an examination of spatial redundancy, which refers to the observation that the most informative region in each video frame usually corresponds to a small image patch, whose shape, size and location shift smoothly across frames. Motivated by this phenomenon, we formulate the patch localization problem as a dynamic decision task, and introduce a spatially adaptive video recognition approach, termed AdaFocus. In specific, a lightweight encoder is first employed to quickly process the full video sequence, whose features are then utilized by a policy network to identify the most task-relevant regions. Subsequently, the selected patches are inferred by a high-capacity deep network for the final prediction. The full model can be trained in end-to-end conveniently. Furthermore, AdaFocus can be extended by further considering temporal and sample-wise redundancies, i.e., allocating the majority of computation to the most task-relevant frames, and minimizing the computation spent on relatively "easier" videos. Our resulting approach, Uni-AdaFocus, establishes a comprehensive framework that seamlessly integrates spatial, temporal, and sample-wise dynamic computation, while it preserves the merits of AdaFocus in terms of efficient end-to-end training and hardware friendliness. In addition, Uni-AdaFocus is general and flexible as it is compatible with off-the-shelf efficient backbones (e.g., TSM and X3D), which can be readily deployed as our feature extractor, yielding a significantly improved computational efficiency. Empirically, extensive experiments based on seven benchmark datasets and three application scenarios substantiate that Uni-AdaFocus is considerably more efficient than the competitive baselines.

Figures (17)

• Figure 1: Comparisons between existing temporal-based methods and our proposed approaches. Most existing works aim to reduce computational costs by selecting a few informative frames to process. Orthogonal to them, AdaFocus reveals that a superior computational efficiency can be achieved by reducing the spatial redundancy. Built upon this finding, we further demonstrate that it is feasible to formulate flexible and highly efficient spatial-temporal dynamic computation simultaneously in a unified framework (Uni-AdaFocus), i.e., attending to the most valuable spatial regions of the most task-relevant video frames.
• Figure 2: Overview of AdaFocus. It first takes a quick glance at each frame $\bm{v}_t$ using a lightweight global encoder $f_{\textnormal{G}}$. Then a policy network $\pi$ is built on top of $f_{\textnormal{G}}$ to select the most important image region $\tilde{\bm{v}}_t$ in terms of recognition. A high-capacity local encoder $f_{\textnormal{L}}$ is adopted to extract features from $\tilde{\bm{v}}_t$. Finally, a classifier aggregates the features across frames to obtain the prediction $\bm{p}_t$.
• Figure 3: Illustration of the policy network $\pi$ in AdaFocusV1. The outputs of $\pi$ parameterize a categorical distribution $\pi(\cdot|{\bm{e}}^{\textnormal{G}}_{1},\ldots, {\bm{e}}^{\textnormal{G}}_{t})$ on multiple patch candidates (here we take 25 as an example). During training, we sample $\tilde{\bm{v}}_t$ from $\pi(\cdot|{\bm{e}}^{\textnormal{G}}_{1},\ldots, {\bm{e}}^{\textnormal{G}}_{t})$, while at test time, we directly select the patch with the largest softmax probability.
• Figure 4: Illustration of interpolation-based patch selection. This operation is differentiable, i.e., the gradients can be directly back-propagated into the policy network $\pi$ through the selected image patch $\tilde{\bm{v}}_t$. Consequently, integrating the learning of $\pi$ into a unified end-to-end training paradigm turns out to be feasible.
• Figure 5: Guiding the training of ${\pi}$ with deep features. The gradients for $\pi$ are obtained by minimizing the deep-feature-based loss $\mathcal{L}_{\pi}^{\textnormal{spatial}}$, instead of being back propagated from the pixel space. With this design, the supervision signals for ${\pi}$ contain much more semantic-level information, which contributes to learning a more effective patch selection policy.