Visual Intention Grounding for Egocentric Assistants

TL;DR

The work addresses egocentric visual intention grounding by introducing EgoIntention, the first dataset that links first-person images with multiple intention queries per object and provides alternative bounding boxes to reflect affordance-based choices. It identifies limitations of off-the-shelf reasoning + grounding pipelines and proposes Reason-to-Ground (RoG) instruction tuning, which disentangles intention reasoning from object grounding and is model-agnostic. Through comprehensive experiments combining RefCOCO/+/g with EgoIntention and LoRA fine-tuning across several LVLMs, RoG achieves significant improvements over naive baselines and outperforms a GPT-4 + GroundingDINO pipeline on EgoIntention, while preserving traditional visual grounding capabilities. The results support a unified approach to visual grounding that handles both explicit object queries and implicit human intentions in both exocentric and egocentric contexts, with practical implications for egocentric assistive systems.

Abstract

Visual grounding associates textual descriptions with objects in an image. Conventional methods target third-person image inputs and named object queries. In applications such as AI assistants, the perspective shifts -- inputs are egocentric, and objects may be referred to implicitly through needs and intentions. To bridge this gap, we introduce EgoIntention, the first dataset for egocentric visual intention grounding. EgoIntention challenges multimodal LLMs to 1) understand and ignore unintended contextual objects and 2) reason about uncommon object functionalities. Benchmark results show that current models misidentify context objects and lack affordance understanding in egocentric views. We also propose Reason-to-Ground (RoG) instruction tuning; it enables hybrid training with normal descriptions and egocentric intentions with a chained intention reasoning and object grounding mechanism. RoG significantly outperforms naive finetuning and hybrid training on EgoIntention, while maintaining or slightly improving naive description grounding. This advancement enables unified visual grounding for egocentric and exocentric visual inputs while handling explicit object queries and implicit human intentions.

Figures (8)

• Figure 1: Traditional visual grounding (left) vs. egocentric visual intention understanding (center and right). Traditional grounding identifies the "white chair" by detecting specific objects from third-person perspectives. Egocentric visual intention understanding must infer user needs in complex, first-person scenarios, e.g. seating in a workshop (center) or using a chair to reach the sink (right).
• Figure 2: Challenge of Visual Intention Grounding. The model must infer the intended object from the full intention sentence, rather than simply detecting explicitly mentioned objects. In this example, "gather my phone and belongings" explicitly mentions "phone" (highlighted in red) , which often misleads existing visual grounding models to identify the wrong object (red box). The correct target, a handbag (green box), is only implied.
• Figure 3: Dataset collection pipeline for EgoIntention. Our data collection process consists of three key stages. (1) Intention Sentence Generation: We use GPT-4 to generate egocentric human intention sentences based on visual input, covering both context-aware and uncommon intention scenarios. (2) Human Validation via MTurk: Annotators assess the semantic validity and real-world applicability of generated sentences, filtering out low-quality or ambiguous descriptions. (3) Alternative Object and Bounding Box Collection: Given the inherent subjectivity of human intentions, additional valid object candidates are identified by human annotators, supplementing the original ground truth annotations with alternative bounding boxes.
• Figure 4: Overview of Reason-to-Ground Instruction Tuning (RoG): The model first infers an explicit object category from an implicit intention sentence (intention reasoning), then localizes the object in the image (object grounding).
• Figure A: Word cloud visualizations of object affordances under different intention contexts. (a) shows word clouds for the object "handbag", where context-aware intentions highlight its primary function of carrying essentials, while uncommon intentions repurpose it as a cushion for a hard bench. (b) presents word clouds for "soap", commonly associated with handwashing or dish cleaning, but also serving an uncommon purpose—making a stuck lock slippery for easier unlocking.