On the Efficacy of Text-Based Input Modalities for Action Anticipation

TL;DR

This work tackles action anticipation by introducing M-CAT, a video transformer that fuses multi-modal signals with text descriptions of actions and objects. It adopts a two-stage process: (i) contrastive pre-training aligning fused multi-modal embeddings with rich, LLM-generated text descriptions of future actions, and (ii) supervised action anticipation fine-tuning using the learned representations. Across EK100, EK55, and EGTEA+ datasets, text-based descriptions improve anticipatory accuracy, particularly for tail and unseen classes, while ablations highlight the contributions of text modalities, GPT-derived descriptions, and self-supervised losses. The approach demonstrates that text-based contextual cues can enhance action anticipation without resorting to modality-specific encoders, suggesting avenues for broader cross-modal pretraining in future work.

Abstract

Anticipating future actions is a highly challenging task due to the diversity and scale of potential future actions; yet, information from different modalities help narrow down plausible action choices. Each modality can provide diverse and often complementary context for the model to learn from. While previous multi-modal methods leverage information from modalities such as video and audio, we primarily explore how text descriptions of actions and objects can also lead to more accurate action anticipation by providing additional contextual cues, e.g., about the environment and its contents. We propose a Multi-modal Contrastive Anticipative Transformer (M-CAT), a video transformer architecture that jointly learns from multi-modal features and text descriptions of actions and objects. We train our model in two stages, where the model first learns to align video clips with descriptions of future actions, and is subsequently fine-tuned to predict future actions. Compared to existing methods, M-CAT has the advantage of learning additional context from two types of text inputs: rich descriptions of future actions during pre-training, and, text descriptions for detected objects and actions during modality feature fusion. Through extensive experimental evaluation, we demonstrate that our model outperforms previous methods on the EpicKitchens datasets, and show that using simple text descriptions of actions and objects aid in more effective action anticipation. In addition, we examine the impact of object and action information obtained via text, and perform extensive ablations.

Figures (6)

• Figure 1: Anticipating actions$\tau_{a}$ seconds after observing information for $\tau_{o}$ seconds using multiple modalities.
• Figure 2: Left: Our training comprises two stages: first, contrastive pre-training, where we fuse embeddings from different modalities using a Fusion Module $\mathcal{F}$, followed by an anticipation module $\mathcal{B}$. The output is contrasted against the rich descriptions of future actions generated using a LLM. The second stage involves fine-tuning a linear layer to predict future actions. Right: Illustration of the image-text and image-image contrastive setup.
• Figure 3: Descriptions generated using the ChatGPT API for actions in the EPIC-Kitchen dataset. The generated descriptions add more contextual cues for the model to learn from. For instance, for the action take chopsticks, the description is already alluding to the future action of "picking up food" or "eating". During training, we randomly select one description for every action.
• Figure 4: Impact of action recognition accuracy on the prediction of verbs, nouns and actions for EK100. Values in dashed lines are the corresponding results from the AFFT baseline.
• Figure 5: Prompt provided to ChatGPT to pick most plausible future action.