Don't Judge by the Look: Towards Motion Coherent Video Representation

TL;DR

This paper tackles overfitting in video understanding by challenging the conventional avoidance of Hue Jittering, showing that hue variance can actually improve motion-based reasoning. It introduces Motion Coherent Augmentation (MCA), which combines SwapMix for efficient RGB-space appearance changes with Variation Alignment to mitigate distribution shift, training with a combined loss $\mathcal{L} = \mathcal{L}_{CE} + \lambda_{AV} \mathcal{L}_{AV}$. Empirically, MCA delivers consistent performance gains across multiple architectures and datasets (Something-Something, UCF101, HMDB51, Kinetics400), and VA can extend improvements to other augmentations like AugMix and RandAugment. This approach enables appearance-invariant representations that emphasize motion, offering a practical, generalizable augmentation strategy for advancing video understanding, albeit with higher memory and training-time requirements.

Abstract

Current training pipelines in object recognition neglect Hue Jittering when doing data augmentation as it not only brings appearance changes that are detrimental to classification, but also the implementation is inefficient in practice. In this study, we investigate the effect of hue variance in the context of video understanding and find this variance to be beneficial since static appearances are less important in videos that contain motion information. Based on this observation, we propose a data augmentation method for video understanding, named Motion Coherent Augmentation (MCA), that introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances. Concretely, we propose an operation SwapMix to efficiently modify the appearance of video samples, and introduce Variation Alignment (VA) to resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representations. Comprehensive empirical evaluation across various architectures and different datasets solidly validates the effectiveness and generalization ability of MCA, and the application of VA in other augmentation methods. Code is available at https://github.com/BeSpontaneous/MCA-pytorch.

Figures (18)

• Figure 1: Illustration of the effect caused by Hue Jittering (denoted by *). Hue Jittering will lead to unrealistic and confusing appearances which are detrimental to object recognition. However, it can improve the performance in video recognition where static appearances are less important.
• Figure 2: Illustration of Motion Coherent Augmentation (MCA). During training, given input video $x$, we will modify its appearance by SwapMix to get $\tilde{x}$. We feed the input pair into the deep network $M\left (\cdot \right )$ to obtain corresponding predictions $p$ and $\tilde{p}$. $p$ will be used to calculate CE loss and $\tilde{p}$ will be encouraged to align with $p$ to enforce the model to learn appearance invariant representation and resolve the distribution shift caused by SwapMix. During inference, only $x$ will be used.
• Figure 3: Illustration of SwapMix. Given a video $x = \left \{ f_{1},f_{2},...,f_{T} \right \}$, its channel order will be shuffled to create the video with a new appearance ${x}'$. Then, interpolation between $x$ and $x^{'}$ will be implemented to generate $\tilde{x}$ with an enlarged input space. The coefficient $\lambda$ is sampled from the Beta distribution to control the degree of interpolation.
• Figure 4: Running time comparisons of Hue Jittering and SwapMix on CPU (Intel(R) Core(TM) i7-6850K) and GPU (NVIDIA GeForce GTX TITAN X). Results are averaged over 500 runs.
• Figure 5: Evaluation of TSM lin2019tsm on different Something-Something V1 validation sets, where the data is original videos, and videos generated by SwapMix, Hue Jittering.