DEVIAS: Learning Disentangled Video Representations of Action and Scene

TL;DR

An encoder-decoder architecture to learn disentangled action and scene representations with a single model and shows favorable performance in various downstream tasks: Diving48, Something-Something-V2, UCF-101, and ActivityNet.

Abstract

Video recognition models often learn scene-biased action representation due to the spurious correlation between actions and scenes in the training data. Such models show poor performance when the test data consists of videos with unseen action-scene combinations. Although scene-debiased action recognition models might address the issue, they often overlook valuable scene information in the data. To address this challenge, we propose to learn DisEntangled VIdeo representations of Action and Scene (DEVIAS), for more holistic video understanding. We propose an encoder-decoder architecture to learn disentangled action and scene representations with a single model. The architecture consists of a disentangling encoder (DE), an action mask decoder (AMD), and a prediction head. The key to achieving the disentanglement is employing both DE and AMD during training time. The DE uses the slot attention mechanism to learn disentangled action and scene representations. For further disentanglement, an AMD learns to predict action masks, given an action slot. With the resulting disentangled representations, we can achieve robust performance across diverse scenarios, including both seen and unseen action-scene combinations. We rigorously validate the proposed method on the UCF-101, Kinetics-400, and HVU datasets for the seen, and the SCUBA, HAT, and HVU datasets for unseen action-scene combination scenarios. Furthermore, DEVIAS provides flexibility to adjust the emphasis on action or scene information depending on dataset characteristics for downstream tasks. DEVIAS shows favorable performance in various downstream tasks: Diving48, Something-Something-V2, UCF-101, and ActivityNet. The code is available at https://github.com/KHU-VLL/DEVIAS.

Figures (8)

• Figure 1: Why do we need disentangled video representations? Video recognition models often learn scene-biased representations due to the spurious correlation li2018resoundwhycantchoi between action and scene in the dataset. Although such models might work well when video contains an action-scene combination seen during training e.g.(a), they would fail when video contains an unseen combination e.g.(b). In contrast, scene debiased video models BEfame could accurately recognize the action regardless of combinations. However, they are not capable of predicting scenes. In this work, we propose to learn disentangled action and scene representations. The disentangled model can understand both action and scene well, including both seen and unseen action-scene combinations, e.g. it correctly predicts dancing on a stage (a) and dancing in a football field (b).
• Figure 2: Overview of DEVIAS. (a) DEVIAS consists of i) a disentangling encoder (DE), ii) an action mask decoder (AMD), and iii) a classification head. (b) Given an input video, the DE first extracts a feature vector using a backbone encoder. Then the DE learns multiple slots. Given input learnable slots as queries, the slot attention iteratively attends to encoded features as keys and values. As a result of the slot attention, the slots progressively learn distinct information, i.e. action, and scene. Then a matching function assigns each slot an action or a scene slot. We train action/scene slots with corresponding labels. (c) Given an action slot, the AMD predicts action masks to learn disentangled representations As slots are complementary in slot attention, the AMD encourages the DE to learn not only good action but also good scene representations.
• Figure 3: Example frames of the datasets. (a) walking with dog on the grass from HVU Seen diba2020hvu, (b) walking with dog in snowfield from HVU Unseen diba2020hvu, (c) dancing on a golf course from HAT Far chung2022hatdataset, (d) snowfield from HAT Scene-Only chung2022hatdataset, and (e) feeding goats from SCUBA VQGAN-CLIP li2023stillmix.
• Figure 4: Architecture of baselines. All baselines use the same encoder, ViT vit. (a) and (b) use separate tokens for action and scene, and (c) and (d) use a single token. (a) and (c) use separate classification heads for action and scene, and (b) and (d) use a unified classification head.
• Figure 5: Slot assignments frequency on UCF-101-SCUBA-VQGAN-CLIP We demonstrate the frequency of the slot assignment as either the action slot or the scene slot. In this experiment, we have two slots before assignment ($K=2$): 'Slot 1' and 'Slot 2'. The result demonstrates that each slot performs a singular role, and the representations of the slots are well-disentangled.