InCoM: Intent-Driven Perception and Structured Coordination for Whole-Body Mobile Manipulation

TL;DR

InCoM infers latent motion intent to dynamically reweight multi-scale perceptual features, enabling stage-adaptive allocation of perceptual attention in whole-body mobile manipulation, and incorporates a geometric-semantic structured alignment mechanism that enhances multimodal correspondence.

Abstract

Whole-body mobile manipulation is a fundamental capability for general-purpose robotic agents, requiring both coordinated control of the mobile base and manipulator and robust perception under dynamically changing viewpoints. However, existing approaches face two key challenges: strong coupling between base and arm actions complicates whole-body control optimization, and perceptual attention is often poorly allocated as viewpoints shift during mobile manipulation. We propose InCoM, an intent-driven perception and structured coordination framework for whole-body mobile manipulation. InCoM infers latent motion intent to dynamically reweight multi-scale perceptual features, enabling stage-adaptive allocation of perceptual attention. To support robust cross-modal perception, InCoM further incorporates a geometric-semantic structured alignment mechanism that enhances multimodal correspondence. On the control side, we design a decoupled coordinated flow matching action decoder that explicitly models coordinated base-arm action generation, alleviating optimization difficulties caused by control coupling. Without access to privileged perceptual information, InCoM outperforms state-of-the-art methods on three ManiSkill-HAB scenarios by 28.2%, 26.1%, and 23.6% in success rate, demonstrating strong effectiveness for whole-body mobile manipulation.

Figures (9)

• Figure 1: Dynamic perceptual attention during the whole-body mobile manipulation. The left half is the color image, and the right half is a schematic diagram of perceptual attention. During manipulation, perceptual attention is primarily focused on local interaction targets; for example, the agent should attend to whether the robotic arm has successfully grasped the trash bin(top). During navigation, perceptual attention shifts toward understanding the global structure to identify freespace (bottom).
• Figure 2: Overview of InCoM. The framework integrates intent-driven multi-scale perception (IDPPM), dual-stream cross-modal affinity refinement (DARM), and decoupled flow-based action generation (DCFM) to produce coordinated whole-body mobile manipulation.
• Figure 3: Comparison of the Action Decoders. (a) Shared Decoder: base and arm actions are jointly modeled by a single decoder with a unified output head. (b) Sequential Hierarchical Decoder: base and arm actions are predicted by independent decoders, where the arm decoder is conditioned on the base output, capturing only unidirectional dependency. (c) DCFM Decoder: base and arm actions are decoded in parallel and exchange information bidirectionally via cross-attention in intermediate layers. To avoid unstable gradient interference during training, stop-gradient is applied in the cross-attention, retaining only forward conditional information.
• Figure 4: Variation of multi-scale modulation weights in the IDPPM during task execution.
• Figure 5: Execution trajectories of our method across four TidyHouse and PrepareGroceries tasks.