Vision and Intention Boost Large Language Model in Long-Term Action Anticipation

TL;DR

We address long-term action anticipation by integrating rich visual cues with high-level behavioral intentions and large language models. The Intention-Conditioned Vision-Language (ICVL) framework infers intentions from video via a vision-language model, fuses them with visual embeddings through the Intention-Context Attention Fusion module, and feeds intention-enriched visuals into a fine-tuned LLM using carefully crafted multi-modal prompts. An effective multi-modality in-context learning strategy selects informative demonstrations from both visual and textual modalities to guide the LLM. Comprehensive experiments on Ego4D, EPIC-Kitchens-55, and EGTEA Gaze+ show state-of-the-art performance, validating that intention-aware fusion and multimodal exemplars consistently improve long-horizon action anticipation in real-world benchmarks.

Abstract

Long-term action anticipation (LTA) aims to predict future actions over an extended period. Previous approaches primarily focus on learning exclusively from video data but lack prior knowledge. Recent researches leverage large language models (LLMs) by utilizing text-based inputs which suffer severe information loss. To tackle these limitations single-modality methods face, we propose a novel Intention-Conditioned Vision-Language (ICVL) model in this study that fully leverages the rich semantic information of visual data and the powerful reasoning capabilities of LLMs. Considering intention as a high-level concept guiding the evolution of actions, we first propose to employ a vision-language model (VLM) to infer behavioral intentions as comprehensive textual features directly from video inputs. The inferred intentions are then fused with visual features through a multi-modality fusion strategy, resulting in intention-enhanced visual representations. These enhanced visual representations, along with textual prompts, are fed into LLM for future action anticipation. Furthermore, we propose an effective example selection strategy jointly considers visual and textual similarities, providing more relevant and informative examples for in-context learning. Extensive experiments with state-of-the-art performance on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+ datasets fully demonstrate the effectiveness and superiority of the proposed method.

Figures (4)

• Figure 1: Illustration of different action anticipation methods. (a) Vision-based methods. (b) Text-based methods. (c) Our proposed Intention-Conditioned Vision-Language (ICVL) model.
• Figure 2: Illustration of Intention-Conditioned Vision-Language (ICVL) model. Given a video, we use a VLM, a visual encoder, and an action recognition model to extract behavioral intention, original visual embeddings, and observed action labels respectively. The behavioral intention and visual embeddings are then integrated into the intention-enhanced visual embeddings through our proposed Intention-Context Attention Fusion (ICAF) module, in which visual features serve as the keys (K) and values (V), while textual intention features act as the queries (Q). Then we consider both visual similarity and textual similarity based on observed action labels to select examples from the training set for in-context learning. Finally, the textual prompt—composed of instructions, observed action labels, and selected examples—along with the intention-enhanced visual embeddings, are fed into the LLM to generate predictions for future action sequences.
• Figure 3: Illustration of prompt for LLMs using in-context learning. The prompt is composed of an instruction, selected examples based on multi-modality similarity, observed actions and intention-enhanced visual embeddings.
• Figure 4: Ablation study on the number of the Selected Examples.