Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

TL;DR

This work investigates zero-shot compositional action recognition in video and identifies object-driven shortcuts as a core failure mode caused by extreme sparsity of verb–object supervision and asymmetric learning difficulty between verbs and objects. It introduces RCORE, a simple framework combining VOCAMix augmentation to diversify unseen verb–object compositions and TORC temporal-order regularization to enforce temporally grounded verb representations. Through open-world, unbiased evaluations on Sth-com and EK100-com, RCORE achieves stronger unseen composition accuracy, reduces co-occurrence bias, and yields positive compositional gaps, demonstrating robust compositional generalization beyond independent verb/object predictions. The findings underscore the importance of addressing shortcut learning and using realistic evaluation protocols for progress in compositional video understanding, with practical implications for CVU systems that must generalize to unseen action compositions.

Abstract

We study Compositional Video Understanding (CVU), where models must recognize verbs and objects and compose them to generalize to unseen combinations. We find that existing Zero-Shot Compositional Action Recognition (ZS-CAR) models fail primarily due to an overlooked failure mode: object-driven verb shortcuts. Through systematic analysis, we show that this behavior arises from two intertwined factors: severe sparsity and skewness of compositional supervision, and the asymmetric learning difficulty between verbs and objects. As training progresses, the existing ZS-CAR model increasingly ignores visual evidence and overfits to co-occurrence statistics. Consequently, the existing model does not gain the benefit of compositional recognition in unseen verb-object compositions. To address this, we propose RCORE, a simple and effective framework that enforces temporally grounded verb learning. RCORE introduces (i) a composition-aware augmentation that diversifies verb-object combinations without corrupting motion cues, and (ii) a temporal order regularization loss that penalizes shortcut behaviors by explicitly modeling temporal structure. Across two benchmarks, Sth-com and our newly constructed EK100-com, RCORE significantly improves unseen composition accuracy, reduces reliance on co-occurrence bias, and achieves consistently positive compositional gaps. Our findings reveal object-driven shortcuts as a critical limiting factor in ZS-CAR and demonstrate that addressing them is essential for robust compositional video understanding.

Figures (11)

• Figure 1: Why object-driven shortcuts emerge in compositional video understanding? (a) Co-occurrence bias. Datasets are intrinsically sparse and highly skewed in their verb–object combinations, creating strong co-occurrence priors. Models exploit these priors as a shortcut: once the object is recognized, the model often predicts the most frequent verb paired with it, ignoring temporal evidence. (b) Asymmetric learning difficulty. Objects are visually explicit and easy to recognize from a single frame, whereas verbs require multi-frame temporal reasoning. This imbalance makes object features dominate training signals. Together, these two factors drive models to adopt object-driven shortcuts, hindering genuine compositional generalization. Best viewed with zoom and color.
• Figure 2: Controlled experiments demonstrate object-driven shortcut learning in ZS-CAR. We empirically identify a key failure mode in ZS-CAR—object-driven shortcuts. (a) Objects are easier to learn than verbs. We train a randomly initialized ViT dosovitskiy202vit on a balanced 10 $\times$ 10 verb-object subset from Sth-com li2024c2c. The learning curves show that object accuracy increases much faster than verb accuracy, indicating that objects are easier to learn. (b) Under a strong co-occurrence bias, models exploit object cues as shortcuts, failing to learn robust verb representations. We prepare a perfectly biased training set and two kinds of test splits utilizing the Sth-com li2024c2c dataset: seen compositions (bias-aligned) and unseen compositions (bias-conflict). Both the randomly initialized ViT dosovitskiy202vit and pretrained CLIP radford2021clip exhibit a consistent trend: while unseen object accuracy increases alongside seen performance, unseen verb accuracy concurrently drops below random chance. Best viewed with zoom and color.
• Figure 3: Learning curve of the SOTA model with our diagnostic metrics. We plot the learning curve of C2C li2024c2c trained on Sth-com li2024c2c. We measure the False Seen Prediction (FSP) and False Co-occurrence Prediction (FCP) ratios, and observe that the seen–unseen accuracy gap ($\Delta_{SU}$) correlates strongly with both metrics. There observations indicate that the current SOTA model exhibits overfitting to seen compositions. Best viewed with zoom and color.
• Figure 4: Overview of RCORE. (a) Overview of our proposed RCORE framework. (b) VOCAMix synthesizes plausible yet unseen verb–object compositions while preserving the temporal structure of the primary video. (c) TORC penalizes alignment between original and temporally perturbed feature vectors, enforcing explicit temporal order modeling and reducing object-driven shortcuts.
• Figure 5: Analysis on the effects of RCORE on the Sth-com li2024c2c dataset. (a) RCORE prevents the False Co-occurrence Prediction (FCP) ratio from increasing during training, whereas the baseline shows a clear rise in FCP. As a result, RCORE consistently maintains a smaller seen–unseen accuracy gap ($\Delta_{SU}$) throughout training. (b) The cosine similarity between the original and reversed verb features becomes strongly negative for RCORE as training progresses, indicating improved temporal discriminative capability. In contrast, the baseline maintains a high similarity (0.91), revealing limited temporal sensitivity. (c) On the Temporal subset, RCORE exhibits a substantially larger performance gap between original and temporally shuffled features compared to the baseline, demonstrating that RCORE learns verb representations that depend on temporal dynamics rather than static object cues. Best viewed with zoom and color.