A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognition

TL;DR

This work targets the gap in transformer-based action recognition for motion-sensitive tasks by introducing Explicit Motion Information Mining (EMIM). EMIM incorporates a cost-volume–style affinity matrix $\mathbf{A}$ constructed from a query-based neighborhood in the next frame with displacement offsets $\Delta x$ and $\Delta y$, plus a relative bias, enabling simultaneous appearance aggregation and explicit motion representation $\mathbf{M}$. The module outputs appearance features $\mathbf{F}$ and motion features $\mathbf{M}$, which are fused and propagated through the network, preserving standard contextual cues while enhancing motion modeling. Empirical results on four datasets including Something-Something V1/V2, SSV1/SSV2, and Diving-48 V2 show clear gains on motion-sensitive data and competitive performance on scene-focused data, with extensive ablations validating design choices such as neighborhood size, sampling strategy, and block replacement strategy.

Abstract

Recently, action recognition has been dominated by transformer-based methods, thanks to their spatiotemporal contextual aggregation capacities. However, despite the significant progress achieved on scene-related datasets, they do not perform well on motion-sensitive datasets due to the lack of elaborate motion modeling designs. Meanwhile, we observe that the widely-used cost volume in traditional action recognition is highly similar to the affinity matrix defined in self-attention, but equipped with powerful motion modeling capacities. In light of this, we propose to integrate those effective motion modeling properties into the existing transformer in a unified and neat way, with the proposal of the Explicit Motion Information Mining module (EMIM). In EMIM, we propose to construct the desirable affinity matrix in a cost volume style, where the set of key candidate tokens is sampled from the query-based neighboring area in the next frame in a sliding-window manner. Then, the constructed affinity matrix is used to aggregate contextual information for appearance modeling and is converted into motion features for motion modeling as well. We validate the motion modeling capacities of our method on four widely-used datasets, and our method performs better than existing state-of-the-art approaches, especially on motion-sensitive datasets, i.e., Something-Something V1 & V2. Our project is available at https://github.com/PeiqinZhuang/EMIM .

Figures (4)

• Figure 1: (a) Affinity matrix generation process. The affinity matrix $\mathbf{A}$ is generated by calculating the visual similarity between the query and key tokens. The depth indicates the magnitude of similarity. (b) Relative performance gains of the transformer-based methods over the competitive CNN baseline method, i.e., TDN wang2021tdn. We choose the competitive TDN as the baseline method. Then, we calculate the relative performance gains of various transformer-based methods on Kinetics-400 carreira2017quo and Sthv2 goyal2017something, respectively. Most of the relative performance gains on Kinetics-400 carreira2017quo (scene-related dataset) are smaller than those on Sthv2 goyal2017something (motion-sensitive dataset). It means that the current transformer-based methods achieve inferior performance in the motion-sensitive dataset, which requires elaborate designs for effective motion modeling.
• Figure 2: Comprehensive comparisons between self-attention and cost volume.(a) Typically, the neighborhood in the self-attention module is the whole image, while cost volume is calculated within a small local neighborhood. (b) The set of key tokens is sampled in a query-irrelevant manner in self-attention, while it is sampled from the query-based neighboring area in cost volume. (c) The affinity matrix $\mathbf{A}$ in self-attention is used for contextual aggregation with value tokens, while it is converted directly into motion features for motion representation in cost volume. $\widetilde{\mathbf{A}}$ denotes a softmax normalized affinity matrix, while $g$ denotes a mapping function for feature transformation. More details can be found in the introduction. (Best viewed when zoomed in and in color)
• Figure 3: Explicit Motion Information Mining module (EMIM). We replace the self-attention module with our EMIM in the first transformer block for every two successive transformer blocks. Given an anchor query token $\mathbf{Q}(t,x,y)$ at the position of $(x,y)$ in the $t^{th}$ frame, a set of key candidate tokens would be adaptively sampled from the query-based neighboring area in the $(t+1)^{th}$ frame to construct the affinity matrix. After that, on the one hand, the constructed affinity matrix is utilized for contextual aggregation for appearance modeling, i.e., $\mathbf{F}(t,x,y)$; on the other hand, it will be flattened and converted into motion features via feature transformation for motion modeling, i.e. $\mathbf{M}(t,x,y)$. Finally, these two kinds of complementary features are combined and fed into the following layers. (Better viewed when zoomed in and in color)
• Figure 4: Visualization. To verify the effectiveness of our method in motion modeling, we select two action instances from SSv1 to show regions of interest of models via the Grad-CAM technique selvaraju2017grad. Based on that, we have two important observations. First, for the first few still frames, there are no obvious movements between those frames. Under this circumstance, only a little attention is paid to the first few frames in our method, while a large portion of areas is mistakenly captured by the baseline method. Second, for the remaining quickly moving frames, our method pays finer and more accurate attention to the moving objects, compared with the baseline method. Based on these findings, it is clear that our method shares the powerful capacity of motion sensitivity and motion capturing. (Best viewed when in color)