LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusor

TL;DR

This letter proposes a novel approach that leverages large models to distill functional rearrangement priors and collects diverse arrangement examples using both LLMs and VLMs and distills the examples into a diffusion model, which balances zero-shot generalization with time efficiency.

Abstract

Object rearrangement, a fundamental challenge in robotics, demands versatile strategies to handle diverse objects, configurations, and functional needs. To achieve this, the AI robot needs to learn functional rearrangement priors in order to specify precise goals that meet the functional requirements. Previous methods typically learn such priors from either laborious human annotations or manually designed heuristics, which limits scalability and generalization. In this work, we propose a novel approach that leverages large models to distill functional rearrangement priors. Specifically, our approach collects diverse arrangement examples using both LLMs and VLMs and then distills the examples into a diffusion model. During test time, the learned diffusion model is conditioned on the initial configuration and guides the positioning of objects to meet functional requirements. In this manner, we create a handshaking point that combines the strengths of conditional generative models and large models. Extensive experiments on multiple domains, including real-world scenarios, demonstrate the effectiveness of our approach in generating compatible goals for object rearrangement tasks, significantly outperforming baseline methods.

Figures (9)

• Figure 1: (a): Integrating a large model into the rearrangement pipeline may lead to compatibility issues. (b): Differently, we distill a conditional generative model from the large models, which helps alleviate this issue.
• Figure 2: (a) Data Generation: We construct an autonomous data collection pipeline to obtain arrangement examples, denoted as $\textit{D}_{f} = \{(P^i, C^i)\}_{i=1}^K$, in two stages, collaboratively using an LLM and a VLM. First, we generate initial arrangement examples, $\{(\hat{P^i}, \hat{C^i})\}_{i=1}^K$, by prompting the VLM and extracting object positions via GroundingDino. Then, we refine these examples using the LLM to obtain the final dataset, $\textit{D}_{f}$. (b) Distillation: The collected dataset is distilled into a score-based diffusion model, denoted as $\vb*{\Phi}_{\theta}$, using a score-matching objective. (c) Inference: During test time, we generate goal positions, $P^g$, using the learned diffusion model and rearrange objects from the initial positions, $P^0$, to the goal positions, $P^g$.
• Figure 3: Incompatible issue of VLM-generated layouts. We leverage LLM to remove the redundant items and reposition the remaining ones to align with corresponding functional requirements. To visualize 'LLM-Refined', we cropped the mask of each object from the results generated by VLM, and then moved the object mask into the LLM refined bounding box.
• Figure 4: Quantitative results across four domains, specifically Coverage Score bars for two functional settings (Vanilla and Left-handed) in two scenarios (Dinner table and Office desk). A lower number indicates a smaller deviation between the generated layouts and the ground truth layouts, signifying better performance. The mean and standard deviation are reported for comparison among our methods and baseline algorithms.
• Figure 5: Visualization of test set examples: We randomly pick 2 test examples from each domain. To visualize test examples (i.e., LLM-refined bounding boxes), we crop the mask of each object from the results generated by VLM, and then move the object mask into the LLM refined bounding box.