Embodied Referring Expression Comprehension in Human-Robot Interaction

TL;DR

The paper tackles the challenge of embodied referring expression comprehension in human-robot interaction by introducing Refer360, a large-scale, multimodal dataset collected in indoor and outdoor environments with egocentric and exocentric viewpoints and rich nonverbal cues. It also proposes MuRes, a multimodal guided residual adapter that acts as an information bottleneck to reinforce modality-specific signals when fused with frozen vision-language models. Across ERFE and VQA benchmarks, MuRes consistently improves performance, demonstrating the value of targeted, salience-driven fusion over standard alignment-based approaches. Together, Refer360 and MuRes establish a robust benchmark and a practical adapter mechanism to advance real-world HRI systems.

Abstract

As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.

Figures (7)

• Figure 1: Refer360 data collection setup in human-robot interaction context (left). Here, the person is pointing towards an object while verbally describing it. Interaction frames from three different views (exo, ego, and depth). Highlighting the canonical frames, i.e., frames where the subject precisely points to an object (right).
• Figure 2: Sample canonical frames from Refer360 dataset in three different views: Exo-view (RGB), Ego-view (RGB), and Exo-View (Depth). The first, second, and third rows contain interaction samples from a home, lab, and outdoor location.
• Figure 3: Multimodal Model, MuRes, with the Guided Residual module. Visual and language representations are extracted and projected from a pre-trained VL model. The projected representations are fed into the cross-attention module as the query. The key and value are the original extracted visual and language representations on the residual connection. The output from the cross-attention module and the projection are summed for downstream task learning.
• Figure 4: Qualitative analysis of VisualBERT with and without the proposed guided residual module (MuRes) on two datasets. Incorporating MuRes improves visual question answering by aligning multimodal context in both ScienceQA and A-OKVQA tasks. Subfigure (a) shows qualitative results on questions involving diagrams from the ScienceQA dataset lu2022learn. Subfigure (b) shows model reasoning on ambiguous visual questions from the A-OKVQA dataset schwenk2022okvqa.
• Figure 5: Objects in Refer360 Dataset