FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination

TL;DR

FALCON tackles loco-manipulation by decoupling base locomotion and arm manipulation into two specialized diffusion policies, coordinating them via a frozen vision–language backbone (CLIP). A phase-progress head provides temporally grounded, language-defined task structure, while a coordination-aware contrastive loss sharpens cross-subsystem compatibility. The approach yields superior performance and robustness on long-horizon tasks and demonstrates effective real-robot deployment with RGB-only perception and open-loop base control. This work suggests foundation-model-guided coordination as a scalable path toward modular, generalizable loco-manipulation systems.

Abstract

We present FoundAtion-model-guided decoupled LoCO-maNipulation visuomotor policies (FALCON), a framework for loco-manipulation that combines modular diffusion policies with a vision-language foundation model as the coordinator. Our approach explicitly decouples locomotion and manipulation into two specialized visuomotor policies, allowing each subsystem to rely on its own observations. This mitigates the performance degradation that arise when a single policy is forced to fuse heterogeneous, potentially mismatched observations from locomotion and manipulation. Our key innovation lies in restoring coordination between these two independent policies through a vision-language foundation model, which encodes global observations and language instructions into a shared latent embedding conditioning both diffusion policies. On top of this backbone, we introduce a phase-progress head that uses textual descriptions of task stages to infer discrete phase and continuous progress estimates without manual phase labels. To further structure the latent space, we incorporate a coordination-aware contrastive loss that explicitly encodes cross-subsystem compatibility between arm and base actions. We evaluate FALCON on two challenging loco-manipulation tasks requiring navigation, precise end-effector placement, and tight base-arm coordination. Results show that it surpasses centralized and decentralized baselines while exhibiting improved robustness and generalization to out-of-distribution scenarios.

Figures (8)

• Figure 1: FALCON actively decouples locomotion and manipulation through two modular diffusion policies, coordinated by a vision-language foundation model. The VLM encodes global scene context, proprioceptive states, and goal instructions into a shared latent embedding that conditions both subsystems.
• Figure 2: Overall framework of FALCON. The blue and green regions denote the decoupled manipulator and quadruped diffusion policies, which act in their own observation and control spaces (wrist/body cameras and end-effector pose for the arm; head/body cameras, base velocity, and body height/pitch for the quadruped). The yellow region shows the foundation model module, which aggregates global RGB observations and the language instruction into a shared latent, while capture task phase and progress simultaneously; this task-representative latent is then used to jointly condition both diffusion policies, providing semantic coordination between the two subsystems.
• Figure 3: Overview of the manipulation tasks. The task on the left shows the quadruped–manipulator system approaching the drawer, adjusting its base pose for optimal arm reachability, opening the drawer, and placing a toy inside. The task on the right illustrates a complementary scenario in which the robot navigates to the drawer while holding a toy, places the toy into the already opened drawer, and then closes the drawer using pose-assisted whole-body manipulation. The top row shows synchronized egocentric camera observations for each stage, color-coded to match the corresponding robot pose, while the bottom row depicts the whole-body motions generated by the decoupled locomotion and manipulation policies.
• Figure 4: Initial robot positions and example navigation trajectories for Task 2. The robot begins from multiple starting locations across the workspace and navigates toward the opened drawer. The color-coding indicates the three evaluation regions: green for the left area, orange for the center area, and blue for the right area. Note that the training data are primarily collected from the center region, making the left and right regions predominantly out-of-distribution (OOD) during evaluation.
• Figure 5: Success rates for Task 1 (precise manipulation) and Task 2 (navigation and placement) across different initial positions. Each subplot reports the performance of the baselines (CDP and LatentTOM), ablated variants of our method (FALCON w/o Phase & CL and FALCON w/o CL), and the full FALCON framework. The results show that FALCON consistently achieves the highest success rates across all task settings and initial configurations. The ACT baseline is omitted from this figure because it achieves a zero percent task-level success rate in all experiments; however, its stage-wise performance can be found in Table \ref{['tab:phase_success']}.