CoINS: Counterfactual Interactive Navigation via Skill-Aware VLM

TL;DR

CoINS introduces a two-tiered framework that merges a skill-aware Vision-Language Model (InterNav-VLM) with a reinforcement-learning–driven skill library to enable interactive navigation in cluttered, partially observable environments without global maps. By grounding parametric skill affordances into metric-scale scenes and employing counterfactual reasoning to assess interaction necessity, the system can decide when and which objects to manipulate. The RL-based traversability-oriented manipulation policy and door-opening skills translate high-level plans into robust whole-body control, enabling efficient path clearance across diverse object categories. Extensive simulation and real-world experiments demonstrate improved success rates and long-horizon navigation performance, with strong generalization across embodiments and objects, representing a significant step toward embodiment-aware robotic interaction in unstructured environments.

Abstract

Recent Vision-Language Models (VLMs) have demonstrated significant potential in robotic planning. However, they typically function as semantic reasoners, lacking an intrinsic understanding of the specific robot's physical capabilities. This limitation is particularly critical in interactive navigation, where robots must actively modify cluttered environments to create traversable paths. Existing VLM-based navigators are predominantly confined to passive obstacle avoidance, failing to reason about when and how to interact with objects to clear blocked paths. To bridge this gap, we propose Counterfactual Interactive Navigation via Skill-aware VLM (CoINS), a hierarchical framework that integrates skill-aware reasoning and robust low-level execution. Specifically, we fine-tune a VLM, named InterNav-VLM, which incorporates skill affordance and concrete constraint parameters into the input context and grounds them into a metric-scale environmental representation. By internalizing the logic of counterfactual reasoning through fine-tuning on the proposed InterNav dataset, the model learns to implicitly evaluate the causal effects of object removal on navigation connectivity, thereby determining interaction necessity and target selection. To execute the generated high-level plans, we develop a comprehensive skill library through reinforcement learning, specifically introducing traversability-oriented strategies to manipulate diverse objects for path clearance. A systematic benchmark in Isaac Sim is proposed to evaluate both the reasoning and execution aspects of interactive navigation. Extensive simulations and real-world experiments demonstrate that CoINS significantly outperforms representative baselines, achieving a 17\% higher overall success rate and over 80\% improvement in complex long-horizon scenarios compared to the best-performing baseline

Figures (13)

• Figure 1: Overview. We introduce CoINS, a hierarchical interactive navigation framework that integrates skill-aware VLM reasoning with RL-based skill library of diverse behaviors. Unlike traditional navigation methods that rely on passive obstacle avoidance, CoINS enables the robot to actively reconfigure the environment to clear paths in cluttered scenarios.
• Figure 2: InterNav-VLM in CoINS framework. The VLM reasoning module takes the robot's egocentric RGB observations and embodiment constraints as input to produce high-level interaction and navigation decisions, which are then translated by the skill execution module into precise motion controls for diverse interaction primitives.
• Figure 3: VQA generation process.
• Figure 4: RL-based hierarchical skill library.
• Figure 5: Diverse scenes and objects in the dataset.