MeMSVD: Long-Range Temporal Structure Capturing Using Incremental SVD

TL;DR

This work tackles long-range video understanding by replacing costly attention-based memory interactions with a low-rank memory representation derived from the memory bank's SVD. The proposed MeMSVD projects actor features onto a compact subspace spanned by the top $n_c$ SVD bases and reconstructs from that subspace, enabling an attention-free interaction and efficient online updates via incremental SVD with a forgetting factor. Empirically, MeMSVD matches or surpasses attention-based methods across AVA and Charades while dramatically reducing computation and memory footprint, and it generalizes across backbones such as SlowFast, ACAR-Net, and MViTv2-S. This approach offers a scalable path to long-term video understanding by combining memory compression with online adaptability, and it demonstrates practical impact for large-scale, temporally extended video tasks.

Abstract

This paper is on long-term video understanding where the goal is to recognise human actions over long temporal windows (up to minutes long). In prior work, long temporal context is captured by constructing a long-term memory bank consisting of past and future video features which are then integrated into standard (short-term) video recognition backbones through the use of attention mechanisms. Two well-known problems related to this approach are the quadratic complexity of the attention operation and the fact that the whole feature bank must be stored in memory for inference. To address both issues, we propose an alternative to attention-based schemes which is based on a low-rank approximation of the memory obtained using Singular Value Decomposition. Our scheme has two advantages: (a) it reduces complexity by more than an order of magnitude, and (b) it is amenable to an efficient implementation for the calculation of the memory bases in an incremental fashion which does not require the storage of the whole feature bank in memory. The proposed scheme matches or surpasses the accuracy achieved by attention-based mechanisms while being memory-efficient. Through extensive experiments, we demonstrate that our framework generalises to different architectures and tasks, outperforming the state-of-the-art in three datasets.

Figures (3)

• Figure 1: Top: Attention-based approaches for memory banks project the clip features $\mathbf{h}_{t}$ into a memory bank through learnable projections. Bottom: We propose to compress the memory bank using SVD by projecting and reconstructing the features $\mathbf{h}_{t}$ into the subspace spanned by the memory basis.
• Figure 2: Ablation Experiments. We conduct ablation on (a): temporal support of the memory features, (b): number of components of the memory representation, and (c): forgetting factor. As backbone we have an ACARNet-50 $8 \times 8$ (upper cell) and ACARNet-101 $8 \times 8$ (bottom cell). All results are on conducted on the AVAv2.2 dataset gu2018 with Kinetics-400 kay2017 or Kinetics-700smaira2020 pre-training. The gray row denotes default choice. (mAP in %)
• Figure 3: a) throughput time for the full network and b) for the head only using an MViTv2-S $16 \times 4$li2022 backbone with three different heads; an attention to memory module (blue), MeMSVD (red) and oMeMSVD (green). We assumed a fixed number of 3 actors per clip in the memory. Results are measured on a single NVIDIA 3090 GPU.