Learning Adaptive Cross-Embodiment Visuomotor Policy with Contrastive Prompt Orchestration

TL;DR

This work tackles cross-embodiment visuomotor policy learning by introducing ContrAstive Prompt Orchestration (CAPO), which combines a hybrid contrastive prompt learning phase with an adaptive prompt orchestration phase. By freezing a CLIP backbone and training a diverse pool of domain-specific prompts through visual, temporal-action, and text objectives, CAPO constructs robust, context-aware representations. The adaptive orchestration then dynamically aggregates prompts conditioned on the current observation, enabling efficient policy optimization and superior zero-shot adaptation to unseen illumination and embodiment changes. The approach demonstrates clear gains in sample efficiency and asymptotic performance, including cross-embodiment deployment without fine-tuning, validated on AI2-THOR/FloorPlan21 for ObjectNav tasks. Overall, CAPO offers a practical pathway to robust, adaptable visuomotor policies in diverse embodied settings.

Abstract

Learning adaptive visuomotor policies for embodied agents remains a formidable challenge, particularly when facing cross-embodiment variations such as diverse sensor configurations and dynamic properties. Conventional learning approaches often struggle to separate task-relevant features from domain-specific variations (e.g., lighting, field-of-view, and rotation), leading to poor sample efficiency and catastrophic failure in unseen environments. To bridge this gap, we propose ContrAstive Prompt Orchestration (CAPO), a novel approach for learning visuomotor policies that integrates contrastive prompt learning and adaptive prompt orchestration. For prompt learning, we devise a hybrid contrastive learning strategy that integrates visual, temporal action, and text objectives to establish a pool of learnable prompts, where each prompt induces a visual representation encapsulating fine-grained domain factors. Based on these learned prompts, we introduce an adaptive prompt orchestration mechanism that dynamically aggregates these prompts conditioned on current observations. This enables the agent to adaptively construct optimal state representations by identifying dominant domain factors instantaneously. Consequently, the policy optimization is effectively shielded from irrelevant interference, preventing the common issue of overfitting to source domains. Extensive experiments demonstrate that CAPO significantly outperforms state-of-the-art baselines in sample efficiency and asymptotic performance. Crucially, it exhibits superior zero-shot adaptation across unseen target domains characterized by drastic environmental (e.g., illumination) and physical shifts (e.g., field-of-view and rotation), validating its effectiveness as a viable solution for cross-embodiment visuomotor policy adaptation.

Figures (11)

• Figure 1: Illustration of domain sensitivity in visuomotor policy learning and the adaptability of the proposed CAPO approach. (a) Conventional learning-based policies are highly sensitive to illumination. An agent trained in a standard environment fails to adapt when facing drastic illumination shifts, even if the underlying scene layout remains unchanged. (b) Similarly, standard approaches struggle with embodiment variations. A policy trained on a specific robot configuration fails when deployed on a different embodiment (e.g., varying FOV). (c) In contrast, CAPO synergizes representation learning and policy optimization via the adaptive orchestration of contrastive prompts. This design enables superior zero-shot adaptability, allowing the agent to successfully navigate in unseen target domains with joint illumination and embodiment variations.
• Figure 2: The framework of contrastive prompt orchestration (CAPO). (a) Contrastive Prompt Learning. We employ a hybrid contrastive learning strategy (incorporating visual, temporal action, and text objectives) to learn a pool of domain-specific visual prompts $\mathbf{P}$ for the frozen CLIP encoder $\Phi$. (b) Adaptive Prompt orchestration. We introduce an adaptive prompt orchestration mechanism that dynamically aggregates these frozen prompts based on the observation $o_t$, generating context-aware features for the policy network $\pi_{\psi}$. The trained policy achieves zero-shot cross-embodiment adaptation in unseen domains by automatically re-weighting prompts to handle visual interference and embodiment variations without fine-tuning.
• Figure 3: Architecture of the adaptive prompt orchestration mechanism. The frozen CLIP encoder extracts features from observation $o_t$ using the prompt pool $\mathbf{P}$. A dual-branch attention mechanism (combining learnable MLP projections and cosine similarity) dynamically re-weights the domain-specific features based on context. Crucially, the text-prompted feature bypasses this attention module and is fused directly. This design explicitly preserves goal-oriented semantics, which are invariant to domain factors, before feeding the policy network $\pi_{\psi}$.
• Figure 4: Training curves of episodic reward for all baselines. All results are averaged over three random seeds, with shaded regions indicating confidence intervals. CAPO demonstrates superior sample efficiency and asymptotic performance, converging faster and achieving higher rewards than all baselines.
• Figure 5: Visualization of navigation episodes for the goal "Bowl" under diverse domain shifts. The columns display egocentric frame sequences from CAPO, ConPE, and PPO across three scenarios: Normal, Illumination Change, and Cross-Embodiment (Smaller FOV). CAPO rapidly identifies and approaches the target in all conditions, demonstrating robust zero-shot adaptation. ConPE exhibits partial adaptability but suffers from delayed target lock-on, requiring more steps to complete the task. In contrast, PPO fails to adapt to environmental interference, resulting in disorientation and task failure.

Theorems & Definitions (2)

• Theorem 1
• proof