Flatten: Video Action Recognition is an Image Classification task

TL;DR

A novel video representation architecture, Flatten, is introduced, which serves as a plug-and-play module that can be seamlessly integrated into any image-understanding network for efficient and effective 3D temporal data modeling.

Abstract

In recent years, video action recognition, as a fundamental task in the field of video understanding, has been deeply explored by numerous researchers.Most traditional video action recognition methods typically involve converting videos into three-dimensional data that encapsulates both spatial and temporal information, subsequently leveraging prevalent image understanding models to model and analyze these data. However,these methods have significant drawbacks. Firstly, when delving into video action recognition tasks, image understanding models often need to be adapted accordingly in terms of model architecture and preprocessing for these spatiotemporal tasks; Secondly, dealing with high-dimensional data often poses greater challenges and incurs higher time costs compared to its lower-dimensional counterparts.To bridge the gap between image-understanding and video-understanding tasks while simplifying the complexity of video comprehension, we introduce a novel video representation architecture, Flatten, which serves as a plug-and-play module that can be seamlessly integrated into any image-understanding network for efficient and effective 3D temporal data modeling.Specifically, by applying specific flattening operations (e.g., row-major transform), 3D spatiotemporal data is transformed into 2D spatial information, and then ordinary image understanding models are used to capture temporal dynamic and spatial semantic information, which in turn accomplishes effective and efficient video action recognition. Extensive experiments on commonly used datasets (Kinetics-400, Something-Something v2, and HMDB-51) and three classical image classification models (Uniformer, SwinV2, and ResNet), have demonstrated that embedding Flatten provides a significant performance improvements over original model.

Figures (6)

• Figure 1: The Flatten operation converts image sequences into single-frame two-dimensional images, enabling the processing of video-understanding tasks like reading a comic strip.
• Figure 2: An overview of action recognition network embedding Flatten operation, which converts a sequence of images into a single-frame image, enabling image-understanding models to be easily applied to video-understanding tasks. Meanwhile, we provide three examples of Flatten operations: (a) row-major transformation, (b) nested transformation, and (c) random transformation.
• Figure 3: Heatmap visualization comparison. Figure \ref{['fig:heatmap']} shows how the Uniformer(2D)-S+Flatten model performs temporal modeling in space. From the figure, we can clearly see that the Uniformer(2D)-S+Flatten model has a more sensitive observation of action information and stronger attention compared to the original Uniformer(3D)-S.
• Figure 4: Optimization curves of Uniformer(3D)-S and Uniformer(2D)-S+Flatten trained on the Kinetics400 dataset. Compared to Uniformer(3D)-S, Uniformer(2D)-S+Flatten demonstrates faster convergence and lower loss values under the same number of training epochs.
• Figure 5: Heatmap visualization. The figure shows a comparison of the attention maps between the Uniformer-S+Flatten and Uniformer (3D)-S models."