PALM: Predicting Actions through Language Models

TL;DR

PALM introduces a three-module framework that bridges visual and textual understanding for long-term action anticipation in egocentric video. By converting past video content into narrations and verb–noun action sequences via a vision-language model and an action recognizer, and then prompting a large language model with MMR-selected exemplars, PALM predicts future action sequences with improved accuracy. The approach achieves state-of-the-art results on the Ego4D LTA benchmark and demonstrates generalization to EK-55 and EGTEA, highlighting the value of grounded, context-rich prompts and exemplar-based in-context learning. Overall, PALM showcases how foundation models can be effectively composed to tackle complex video understanding tasks with limited direct video data.

Abstract

Understanding human activity is a crucial yet intricate task in egocentric vision, a field that focuses on capturing visual perspectives from the camera wearer's viewpoint. Traditional methods heavily rely on representation learning that is trained on a large amount of video data. However, a major challenge arises from the difficulty of obtaining effective video representation. This difficulty stems from the complex and variable nature of human activities, which contrasts with the limited availability of data. In this study, we introduce PALM, an approach that tackles the task of long-term action anticipation, which aims to forecast forthcoming sequences of actions over an extended period. Our method PALM incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. By leveraging the context provided by these past events, we devise a prompting strategy for action anticipation using large language models (LLMs). Moreover, we implement maximal marginal relevance for example selection to facilitate in-context learning of the LLMs. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation on the Ego4D benchmark. We further validate PALM on two additional benchmarks, affirming its capacity for generalization across intricate activities with different sets of taxonomies.

Figures (12)

• Figure 1: PALM for long-term action anticipation. Given a video input, PALM uses a vision-language model (VLM) and an action recognition model (ARM) to compile a text prompt that describes the past video content. A large-language model (LLM) then takes the prompt and predicts future actions.
• Figure 2: Overview of PALM. Given an input video, an image captioning model and an action recognition model convert visual information of past activities into text format (visual context and past actions). Then, LLM utilizes visual context and past actions to predict the sequence of future actions. The image captioning model is frozen and in-context learning is adapted for LLM prompting.
• Figure 3: List of questions asking for information related to key concepts of egocentric image.
• Figure 4: Action recognition network. A frozen video encoder generates visual features from $n=4$ continuous action segments, followed by transformer layers and classification heads.
• Figure 5: Prompt structure for LLMs. The prompt is composed of an instruction, examples retrieved from the training set, and the query related to the prediction video. Each video description consists of narrations, past actions, and future actions.