SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction

TL;DR

SInViG introduces a self-evolving framework for interactive visual grounding in human-robot interaction, leveraging large-scale unlabeled images and LLM-polished dialogues to automatically generate training data and iteratively improve five tasks across IVG benchmarks and real-robot manipulation. The architecture unifies a tokenizer, vision encoder, and auto-regressive transformer to support the Questioner, Guesser, and Oracle within a single model, trained with cross-entropy losses on multi-turn interactions. Empirically, SInViG achieves state-of-the-art results on the InViG benchmark, demonstrates strong performance in human-robot interaction with volunteers, and maintains robust real-robot manipulation capabilities on a Kinova Gen3 platform, underscoring practicality for open-world HRI. A key insight is that the combination of abundant unlabeled visuals and LLM-driven data polishing yields meaningful improvements, while ablations show polishing is essential to avoid degrading language quality, making the approach scalable to diverse environments and users.

Abstract

Linguistic ambiguity is ubiquitous in our daily lives. Previous works adopted interaction between robots and humans for language disambiguation. Nevertheless, when interactive robots are deployed in daily environments, there are significant challenges for natural human-robot interaction, stemming from complex and unpredictable visual inputs, open-ended interaction, and diverse user demands. In this paper, we present SInViG, which is a self-evolving interactive visual agent for human-robot interaction based on natural languages, aiming to resolve language ambiguity, if any, through multi-turn visual-language dialogues. It continuously and automatically learns from unlabeled images and large language models, without human intervention, to be more robust against visual and linguistic complexity. Benefiting from self-evolving, it sets new state-of-the-art on several interactive visual grounding benchmarks. Moreover, our human-robot interaction experiments show that the evolved models consistently acquire more and more preferences from human users. Besides, we also deployed our model on a Franka robot for interactive manipulation tasks. Results demonstrate that our model can follow diverse user instructions and interact naturally with humans in natural language, despite the complexity and disturbance of the environment.

Figures (12)

• Figure 1: Overview of SInViG. SInViG is a self-evolving model for interactive visual grounding. Left: self-evolving training iteration; Right: the demonstration of inference steps for SInViG.
• Figure 2: Performance of SInViG with different self-evolving iteration rounds. (a) The comparison across models with different rounds of self-evolving iteration. (b) Ablation study of SInViG with/without LLM polishment & additional image sources. Note that lower is better for Rank, and higher is better for other metrics. Metrics except for Rank are normalized (divided) by the maximum value in each group for visualization.
• Figure 3: Results of human-robot experiments. Top row: success rate on TiO-HRI bench. Bottom row: scores of human evaluation. Scores are selected from "better" (green), "tie" (blue), and "worse" (red).
• Figure 4: Examples of human-robot interaction. We have shown examples from SInViG-R0 to SInViG-R2. We can see that the SInViG-R2 is more efficient and accurate during interaction. It asks fewer redundant questions and follows instructions from the users more properly, benefiting from the LLM-augmented dialogues. By contrast, SInViG-V0 is more redundant and sometimes asks repeated questions.
• Figure 5: More examples of SInViG in realistic applications. (a) Understanding simple calculations. (b) Multi-modal interaction. (c) Online correction.