PALM: Enhanced Generalizability for Local Visuomotor Policies via Perception Alignment

TL;DR

This work tackles the challenge of generalizing image-based imitation learning for robotic manipulation across workspace, viewpoint, and embodiment shifts. It introduces PALM, which modularizes policies into coarse global actions and a generalizable local policy, and applies visual and proprioceptive input alignment as preprocessing to recover invariant local actions under OOD. The method achieves substantial improvements, reducing OOD degradation to $8 ext{%}$ in simulation and $24 ext{%}$ in the real world (compared to $45 ext{%}$ and $77 ext{%}$ for baselines in some settings), with ablations confirming the importance of both alignment components. PALM's preprocessing-based approach requires no extra modalities, data collection, or model changes and shows promise for extending generalization to multi-stage tasks and cross-domain robotic manipulation.

Abstract

Generalizing beyond the training domain in image-based behavior cloning remains challenging. Existing methods address individual axes of generalization, workspace shifts, viewpoint changes, and cross-embodiment transfer, yet they are typically developed in isolation and often rely on complex pipelines. We introduce PALM (Perception Alignment for Local Manipulation), which leverages the invariance of local action distributions between out-of-distribution (OOD) and demonstrated domains to address these OOD shifts concurrently, without additional input modalities, model changes, or data collection. PALM modularizes the manipulation policy into coarse global components and a local policy for fine-grained actions. We reduce the discrepancy between in-domain and OOD inputs at the local policy level by enforcing local visual focus and consistent proprioceptive representation, allowing the policy to retrieve invariant local actions under OOD conditions. Experiments show that PALM limits OOD performance drops to 8% in simulation and 24% in the real world, compared to 45% and 77% for baselines.

Figures (9)

• Figure 1: Input modalities under OOD shift. Workspace, camera, and embodiment shifts cause significant variations in the third-person image relative to in-domain settings, while changes in the local area near the end-effector remain less pronounced. Corresponding misalignments (in red) also occur in proprioceptive states, e.g., EE $(x, y)$, rotation $\mathbf{R}$, and robot joint angles $\mathbf{J}$.
• Figure 2: Alignments for local policy. PALM performs visual and proprioceptive alignment on the input as data pre-processing for local manipulation policies.
• Figure 3: TCP-centric vs. object-centric cropping at time steps 1, 18, and 50 on Lift Spam Invisible. TCP cropping carries the robot's movement across frames.
• Figure 4: Visual domain shifts in third-person views that cannot be resolved through TCP-centric cropping alone.
• Figure 5: Tasks and OOD setups. Four simulation tasks with testing workspace ranges shown in green and training ranges in other colors. Testing domains for a selected camera viewpoint and cross-embodiment are shown in Fig. \ref{['fig:teasor']}.