Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions

TL;DR

This work introduces intention-driven visual grounding (IVG) to interpret non-literal human intents in open-world, egocentric, multi-scene settings. It defines a two-stage IVG task—intent interpretation with multi-scene perception followed by object grounding—and builds IntentionVG, the largest dataset of free-form intention expressions aligned with egocentric views. Baselines in zero-shot and fine-tuning settings, including LLM-assisted intention interpretation and scene perceivers, demonstrate the necessity of intent understanding for accurate grounding, with substantial gains from data and model integration. The dataset and baselines aim to catalyze research in intention-oriented grounding, with broader implications for embodied AI and human-machine interaction, while noting limitations in scale and segmentation capabilities.

Abstract

Visual grounding (VG) aims at locating the foreground entities that match the given natural language expressions. Previous datasets and methods for classic VG task mainly rely on the prior assumption that the given expression must literally refer to the target object, which greatly impedes the practical deployment of agents in real-world scenarios. Since users usually prefer to provide intention-based expression for the desired object instead of covering all the details, it is necessary for the agents to interpret the intention-driven instructions. Thus, in this work, we take a step further to the intention-driven visual-language (V-L) understanding. To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a large-scale IVG dataset termed IntentionVG with free-form intention expressions. Considering that practical agents need to move and find specific targets among various scenarios to realize the grounding task, our IVG task and IntentionVG dataset have taken the crucial properties of both multi-scenario perception and egocentric view into consideration. Besides, various types of models are set up as the baselines to realize our IVG task. Extensive experiments on our IntentionVG dataset and baselines demonstrate the necessity and efficacy of our method for the V-L field. To foster future research in this direction, our newly built dataset and baselines will be publicly available at https://github.com/Rubics-Xuan/IVG.

Figures (11)

• Figure 1: Task Comparison between Affordance Detection (AD), Task-Driven Object Detection (TDOD), Referring Expression Comprehension (REC), Intention-Oriented Object Detection (IOOD) and Intention-driven VG (IVG).
• Figure 2: The illustration about the overall pipeline of our intention-driven visual grounding task, which mainly comprises intention interpretation, multi-scene perception and the subsequent visual grounding.
• Figure 3: The illustration of data collection engine for IntentionVG. We start by inheriting EgoObjects zhu2023egoobjects data and conduct scene category labeling for each image. Then we feed GPT-4 with V-L input to generate the draft of intention-driven response. At last, we conduct data filtering by manually selecting the well matched expression-bounding box (bbox) pairs.
• Figure 4: IntentionVG dataset statistics. (a) the number of referring expressions per object's category in the log scale. (b) the word cloud highlights the head categories.
• Figure 5: Word clouds of partial categories from our IntentionVG benchmark dataset.