Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning

TL;DR

Robust egocentric visual attention prediction remains challenging due to dynamic, context-rich scenes. The paper introduces a language-guided scene context-aware learning framework with a context perceiver that, guided by language scene descriptions, yields context-aware video representations for PoI prediction. Two training losses, $\mathcal{L}_{neg}$ and $\mathcal{L}_{supp}$, along with a scene-context alignment $\mathcal{L}_{context}$, encourage precise PoI localization and suppression of irrelevant regions, achieving state-of-the-art results on Ego4D and AEA and showing strong generalization to unseen videos. The approach reduces inference-time cost by relying on training-time language guidance, and it opens avenues for extending to other egocentric tasks and multimodal extensions.

Abstract

As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the inherent complexity and ambiguity of dynamic egocentric scenes. Motivated by evidence that scene contextual information plays a crucial role in modulating human attention, in this paper, we present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction. We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description, generating context-aware video representations. We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention. Extensive experiments on Ego4D and Aria Everyday Activities (AEA) datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance and enhanced robustness across diverse, dynamic egocentric scenarios.

Figures (7)

• Figure 1: An example showing how contextual cues help predict the point-of-interest region. When humans observe the given scene (left), humans can understand the scene context--a red bowl with an egg mixture and a whisk in hand. Therefore, humans easily infer that the red bowl will likely become the focus of first-person attention.
• Figure 2: The examples of scene summary descriptions, which include location, action, and object information (e.g., living room, reaching for a remote control, and TV) related with the first person.
• Figure 3: (a) The overall architecture of egocentric visual attention anticipation framework along with the proposed context perceiver. (b) The details of context perceiver consisting of context summary extractor and guider. The context summary extractor is designed to obtain context summary representations by looking through whole video frames, and the context summary guider helps frame region tokens refer to the scene context summary tokens for each frame respectively. During the training, context summary tokens are guided to represent the scene summary description embedding which is paired with the input video.
• Figure 4: The examples of the negative points located around the target point.
• Figure 5: The examples show whether the context perceiver can properly capture scene context representation. When the input video is observed, the context perceiver outputs context summary tokens. Then we compare the similarity between the tokens and summary description embeddings. In the first row, the context summary token is well matched with the corresponding scene summary description showing the highest similarity 0.84.