UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning

TL;DR

UniFormer addresses the challenge of learning efficient, multi-scale spatiotemporal video representations by unifying 3D convolution with spatiotemporal self-attention in a single transformer framework. The key innovation, Multi-Head Relation Aggregator (MHRA), employs local affinities in shallow layers for computational efficiency and global, content-based affinities in deep layers for long-range modeling, aided by Dynamic Position Embedding. This design yields strong performance with substantially reduced GFLOPs on Kinetics-400/600 and Something-Something V1/V2, and demonstrates robust transfer from ImageNet-1K pretraining. The results establish a practical, scalable path for high-quality video understanding that balances accuracy and efficiency.

Abstract

It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at https://github.com/Sense-X/UniFormer.

Figures (8)

• Figure 1: Some visualizations of TimeSformer. We respectively show the feature, spatial and temporal attention from the 3rd layer of TimeSformer timesformer. We find that, such transformer learns local representations with redundant global attention. For an anchor token (green box), spatial/temporal attention compares it with all the contextual tokens for aggregation, while only its neighboring tokens (boxes filled with red color) actually work. Hence, it wastes large computation to encode only very local spatiotemporal representations.
• Figure 2: Accuracy vs. per-video GFLOPs on Kinetics-400 and Something-Something V2. B-32(4) means we test UniFormer-B$_{32f}$ with 4 clips and S-16(3) means we test UniFormer-S$_{16f}$ with 3 crops (more testing details can be found in Section \ref{['ablation_studies']}). Our UniFormer achieves the best balance between accuracy and computation on both datasets.
• Figure 3: Overall architecture of our Unified transFormer (UniFormer). A UniFormer block consists of three key modules, i.e., Dynamic Position Embedding (DPE), Multi-Head Relation Aggregrator (MHRA), and Feed Forward Network (FFN). Detailed explanations can be found in Section \ref{['method']}.
• Figure 4: Multi-clip/crop testing comparison on different datasets. Multi-clip testing is better for Kinetics and multi-crop testing is better for Something-Something.
• Figure 5: hoverboarding.