Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views

TL;DR

The paper presents Look and Tell, a dataset designed to study multimodal grounding across egocentric and exocentric views in a naturalistic kitchen setting. It combines synchronized gaze, speech, egocentric video from Meta Aria glasses, exocentric GoPro footage, and 3D room reconstructions to enable systematic comparisons between 2D and 3D representations and between ego and exo perspectives. An end-to-end annotation pipeline uses WhisperX, GPT-based mention extraction, MolMo object localization, and SAM2 tracking, with manual curation to handle challenging cases, resulting in 2,707 referential mentions and 2,504 gaze–speech linked events. The dataset serves as a benchmark for embodied agents to understand situated dialogue and references in space, facilitating advances in spatial grounding, referential resolution, and human–robot collaboration.

Abstract

We introduce Look and Tell, a multimodal dataset for studying referential communication across egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze, speech, and video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect multimodal grounding. The dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of embodied agents that can understand and engage in situated dialogue.

Figures (11)

• Figure 1: Example of synchronized video feeds. The exocentric (left) view provides situational context, while the egocentric (right) view captures the participant's first-person perspective, including their gaze target (green circle, overlaid for visualization).
• Figure 2: Example of reconstructed 3D room point cloud used as the canonical reference space. All point clouds extracted from Aria recordings are aligned and canonicalized to this shared coordinate system.
• Figure 3: Annotation pipeline for the egocentric camera: synchronized audio and images are processed with WhisperX for word-level transcripts, GPT for object mentions, Molmo for object detection, and SAM2 for mask propagation.
• Figure 4: Examples of annotation strategies: (a) Molmo detection based automated SAM2 mask propagation, (b)manual seeding of tracker with annotation interface.
• Figure : Paprika mention (“slice the paprika”)