Learning to Act Robustly with View-Invariant Latent Actions

TL;DR

VILA targets viewpoint robustness by shifting invariance from static scene features to the dynamics governing actions. It learns a view-invariant latent action space through an action-guided contrastive objective and a global structure alignment, then uses latent behavior cloning to train a view-robust encoder that conditions a downstream visuomotor policy. Empirically, VILA demonstrates superior unseen-view generalization and data-efficient unseen-task adaptation across five simulated tasks and a real-world SO-ARM setup, outperforming scene-level baselines and maintaining performance under extrapolated viewpoints. This dynamics-centered pretraining yields more transferable priors, enabling robust policy transfer with limited new demonstrations and suggesting a promising direction for robust visuomotor learning under camera changes.

Abstract

Vision-based robotic policies often struggle with even minor viewpoint changes, underscoring the need for view-invariant visual representations. This challenge becomes more pronounced in real-world settings, where viewpoint variability is unavoidable and can significantly disrupt policy performance. Existing methods typically learn invariance from multi-view observations at the scene level, but such approaches rely on visual appearance and fail to incorporate the physical dynamics essential for robust generalization. We propose View-Invariant Latent Action (VILA), which models a latent action capturing transition patterns across trajectories to learn view-invariant representations grounded in physical dynamics. VILA aligns these latent actions across viewpoints using an action-guided objective based on ground-truth action sequences. Experiments in both simulation and the real world show that VILA-based policies generalize effectively to unseen viewpoints and transfer well to new tasks, establishing VILA as a strong pretraining framework that improves robustness and downstream learning performance.

Figures (16)

• Figure 1: VILA Overview. Our method learns view-invariant latent actions by aligning them using action-aware contrastive learning along with predicting the future. A latent policy that predicts these latent actions from the current observation is then used as a vision encoder to condition a downstream visuomotor policy, yielding robust view generalization and task adaptation in simulation and the real-world.
• Figure 2: Dataset Overview. All images are displayed at the same scale. Top rows show simulation tasks, and the bottom row shows real-world experiments.
• Figure 3: Real-world Unseen Views. Rows show different tasks: (Top) Pick & Place, (Bottom) Drawer.
• Figure 4: Multi-View Camera Poses for Training and Evaluations.Green poses are used for training the encoder and policy (seen), and red poses are reserved for evaluation (unseen). An extrapolated viewpoint (beyond $5\times5$ grid) is generated starting from the blue pose.
• Figure 5: Unseen View Generalization vs. Viewpoint Difference. Success rates (%) (averaged over 20 episodes per each view) of VILA and baseline methods for view generalization under (a)fine-tuned and (b)frozen encoder settings. The success rates are shown with respect to angular differences from the training viewpoints. We evaluate 15 unseen camera viewpoints and, for each unseen view, compute its view difference to the closest training view among the 10 seen cameras, measured as the Euclidean norm in the (azimuth, elevation) space. Based on this, the 15 unseen views are sorted and partitioned into four groups (of sizes 4, 4, 4, and 3). On the x-axis, we plot the average nearest-view difference within each group, and on the y-axis we report the corresponding average success rate for that group.