Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?

TL;DR

This work questions whether ego-centric video-language models truly grasp hand-object interactions, revealing that current EgoVLMs falter when HOI sentences are subtly altered. It introduces EgoHOIBench to systematically test verb- and noun-level HOI distinctions and uncovers an object-centric bias that underperforms on verb understanding. To remedy this, the authors propose EgoNCE++, an asymmetric pretraining objective with HOI-aware hard-negative generation for video-to-text and object-centric sampling for text-to-video, yielding consistent gains across multiple EgoVLMs and downstream HOI tasks. The approach advances open-vocabulary HOI recognition and temporal understanding while maintaining generalization, with practical implications for robust embodied perception in real-world settings.

Abstract

Egocentric video-language pretraining is a crucial step in advancing the understanding of hand-object interactions in first-person scenarios. Despite successes on existing testbeds, we find that current EgoVLMs can be easily misled by simple modifications, such as changing the verbs or nouns in interaction descriptions, with models struggling to distinguish between these changes. This raises the question: Do EgoVLMs truly understand hand-object interactions? To address this question, we introduce a benchmark called EgoHOIBench, revealing the performance limitation of current egocentric models when confronted with such challenges. We attribute this performance gap to insufficient fine-grained supervision and the greater difficulty EgoVLMs experience in recognizing verbs compared to nouns. To tackle these issues, we propose a novel asymmetric contrastive objective named EgoNCE++. For the video-to-text objective, we enhance text supervision by generating negative captions using large language models or leveraging pretrained vocabulary for HOI-related word substitutions. For the text-to-video objective, we focus on preserving an object-centric feature space that clusters video representations based on shared nouns. Extensive experiments demonstrate that EgoNCE++ significantly enhances EgoHOI understanding, leading to improved performance across various EgoVLMs in tasks such as multi-instance retrieval, action recognition, and temporal understanding. Our code is available at https://github.com/xuboshen/EgoNCEpp.

Figures (17)

• Figure 1: Although EgoVLMs have been pretrained on millions of worldwide egocentric videos and applied to challenging downstream tasks like video-text retrieval, we observe that they often fail to select the matched sentence from the simplest word-substituted candidates for videos.
• Figure 2: EgoHOI performance of EgoVLMs on EgoHOIBench.
• Figure 3: Visualization of LaViLa's feature space. Both video and text feature space exhibits the object-centric property. Apparently, the videos/texts are more separable by nouns, indicating a video is more easily matched with the correct noun on HOI-noun tests rather than verbs.
• Figure 4: Illustration of our pretraining framework. (a) EgoVLMs are trained with EgoNCE++, where the visual encoder is trained using LoRA hu2022lora to enhance video representation, while the text encoder remains frozen. Specifically, EgoNCE++ consists of (b) V2T: generating HOI-related negative captions for fine-grained supervision, and (c) T2V: strengthening the strong ability of EgoVLMs to recognize nouns by aggregating video features associated with similar nouns.
• Figure 5: Overview of experimental results. (a) LaViLa++ that is pretrained on LaViLa using EgoNCE++ achieves remarkable improvements across benchmarks under zero-shot settings, meanwhile (b) EgoNCE++ universally enhances HOI comprehension on EgoHOIBench across EgoVLMs.