Can masking background and object reduce static bias for zero-shot action recognition?

TL;DR

The paper tackles static bias in zero-shot action recognition, showing that CLIP-based models can rely on backgrounds and objects rather than human actions. It introduces masking-based data augmentation to suppress background/object cues, and evaluates on Kinetics400, Mimetics, and SSv2 using ViFi-CLIP and ActionCLIP with dynamic masking strategies. The results demonstrate that background masking reduces background reliance and improves person-focused inference in zero-shot and cross-dataset settings, while object masking reveals strong object cues, especially in SSv2, guiding when and how masking should be applied. Overall, masking approaches improve the focus on human action and offer a practical direction for reducing static bias in zero-shot action recognition with video-language models.

Abstract

In this paper, we address the issue of static bias in zero-shot action recognition. Action recognition models need to represent the action itself, not the appearance. However, some fully-supervised works show that models often rely on static appearances, such as the background and objects, rather than human actions. This issue, known as static bias, has not been investigated for zero-shot. Although CLIP-based zero-shot models are now common, it remains unclear if they sufficiently focus on human actions, as CLIP primarily captures appearance features related to languages. In this paper, we investigate the influence of static bias in zero-shot action recognition with CLIP-based models. Our approach involves masking backgrounds, objects, and people differently during training and validation. Experiments with masking background show that models depend on background bias as their performance decreases for Kinetics400. However, for Mimetics, which has a weak background bias, masking the background leads to improved performance even if the background is masked during validation. Furthermore, masking both the background and objects in different colors improves performance for SSv2, which has a strong object bias. These results suggest that masking the background or objects during training prevents models from overly depending on static bias and makes them focus more on human action.

Figures (6)

• Figure 1:
• Figure 2:
• Figure 4: Training procedure with the masking approaches.
• Figure 5:
• Figure 6: