X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning

TL;DR

X-Distill tackles data scarcity in visuomotor learning by distilling the semantic knowledge of a large Vision Transformer into a compact CNN encoder trained on ImageNet. The distilled encoder is jointly fine-tuned with a diffusion policy head on robotics data, yielding strong data-efficient performance across 34 simulated tasks and 5 real-world tasks. The results surpass baselines based on training from scratch, finetuning ViTs, and even some 3D or Vision-Language-Action models, highlighting the value of domain-agnostic cross-architecture distillation for robotics. This approach offers a practical route to leveraging powerful vision priors while maintaining the sample efficiency required in real-world manipulation tasks.

Abstract

Visuomotor policies often leverage large pre-trained Vision Transformers (ViTs) for their powerful generalization capabilities. However, their significant data requirements present a major challenge in the data-scarce context of most robotic learning settings, where compact CNNs with strong inductive biases can be more easily optimized. To address this trade-off, we introduce X-Distill, a simple yet highly effective method that synergizes the strengths of both architectures. Our approach involves an offline, cross-architecture knowledge distillation, transferring the rich visual representations of a large, frozen DINOv2 teacher to a compact ResNet-18 student on the general-purpose ImageNet dataset. This distilled encoder, now endowed with powerful visual priors, is then jointly fine-tuned with a diffusion policy head on the target manipulation tasks. Extensive experiments on $34$ simulated benchmarks and $5$ challenging real-world tasks demonstrate that our method consistently outperforms policies equipped with from-scratch ResNet or fine-tuned DINOv2 encoders. Notably, X-Distill also surpasses 3D encoders that utilize privileged point cloud observations or much larger Vision-Language Models. Our work highlights the efficacy of a simple, well-founded distillation strategy for achieving state-of-the-art performance in data-efficient robotic manipulation.

Figures (5)

• Figure 1: X-Distill is a simple yet effective visual encoder enabling data-efficient visuomotor learning. A. X-Distill is obtained by cross-architecture knowledge distillation from a large ViT teacher into a compact CNN student on general-purpose image datasets. B. Designed for visuomotor policy learning, X-Distill can be jointly fine-tuned end-to-end with a diffusion policy head on robotics-specific datasets. C. Given a few ($20\sim25$) demonstrations per task, X-Distill significantly outperforms representative counterparts on real-world manipulation tasks, exhibiting its surprising effectiveness.
• Figure 2: Visualization of configurations for our real-world tasks. The orange arrow provides a schematic representation of the gripper trajectory as derived from the data. The green regions represent the distribution of object/robot configurations seen during training demonstrations, while the red regions illustrate the novel configurations used for generalization testing.
• Figure 3: Representative trajectory types observed in the "Writing AGI" task. We identify three distinct behaviors: (1) Ideal Behavior: Successful and robust execution of all three letters, even under perturbation. (2) Repetitive Loop: Perseverative behavior where the policy gets stuck repeatedly writing the first letter 'A'. (3) Persistent Hesitation: Dithering motion above the paper without initiating the writing task.
• Figure 4: t-SNE visualization of learned feature spaces on the "Writing AGI" task. Our X-Distill encoder learns to form three distinct clusters corresponding to the task's semantic stages, quantitatively confirming a well-separated feature space with a high Silhouette Score rousseeuw1987silhouettes of $0.472$, which indicates a high degree of cluster cohesion and separation compared with the baselines. This semantic separability is crucial for the policy to accurately identify the current task stage, enabling precise long-horizon planning for the sequential writing task.
• Figure 5: Saliency map comparison on the "Writing AGI" task. We visualize the model's visual focus at the beginning of each writing stage. Our X-Distill encoder correctly shifts its attention from the gripper (before 'A'), to the letter 'A' (before 'G'), and finally to the letter 'G' (before 'I'). Baseline models exhibit diffuse or irrelevant attention.