Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraints

TL;DR

This work tackles zero-shot object-object alignment of two meshes under a text prompt by optimizing relative pose and scale at test time using differentiable rendering and vision-language supervision. It augments CLIP-based guidance with geometry-aware objectives—fractional soft-ICP to promote surface contact and a penetration penalty to enforce physical plausibility—while employing phased optimization and camera scheduling to balance exploration and refinement. An LLM-guided hyperparameter strategy further tailors scale, contact, and penetration to each scene. A dedicated 50-pair benchmark demonstrates that the proposed method achieves higher semantic alignment with lower interpenetration than baselines, and a user study corroborates improved perceptual plausibility. The approach enables robust, text-driven 3D scene assembly without requiring ground-truth 3D alignment data, with potential applications in image-to-3D alignment and iterative multi-object composition.

Abstract

We study zero-shot 3D alignment of two given meshes, using a text prompt describing their spatial relation -- an essential capability for content creation and scene assembly. Earlier approaches primarily rely on geometric alignment procedures, while recent work leverages pretrained 2D diffusion models to model language-conditioned object-object spatial relationships. In contrast, we directly optimize the relative pose at test time, updating translation, rotation, and isotropic scale with CLIP-driven gradients via a differentiable renderer, without training a new model. Our framework augments language supervision with geometry-aware objectives: a variant of soft-Iterative Closest Point (ICP) term to encourage surface attachment and a penetration loss to discourage interpenetration. A phased schedule strengthens contact constraints over time, and camera control concentrates the optimization on the interaction region. To enable evaluation, we curate a benchmark containing diverse categories and relations, and compare against baselines. Our method outperforms all alternatives, yielding semantically faithful and physically plausible alignments.

Figures (24)

• Figure 1: Text-guided object-object alignment and iterative composition. The figure shows four independent examples, each presenting the input meshes and text prompt alongside our alignment result. In addition, an iterative example demonstrates progressive assembly of a burger: the output of stage $k$ is incorporated into the input of stage $k{+}1$, gradually forming the final arrangement.
• Figure 2: Overview of the proposed pipeline. Given two meshes and a text prompt, we optimize the relative pose and scale to produce a text-consistent alignment over $P$ phases. In each phase, we compose the scene, render with a differentiable renderer to obtain a semantic loss, and compute geometric losses. The best result of phase $i$ initializes phase $i{+}1$; across phases we increase the fractional soft-ICP and penetration weights and progressively zoom the cameras in. The final output is an aligned 3D placement of the two meshes.
• Figure 3: Effect of the alignment ratio $r$ on a grilled-toast pair. The two objects are optimized with the same prompt, "grilled cheese toasts", while varying $r$. With $r{=}1.0$, the top toast aligns directly above the bottom toast, producing broad surface contact; as $r$ decreases, attachment is encouraged over a smaller subset of vertices and the contact region correspondingly shrinks.
• Figure 4: Penetration loss geometry. For target vertex $v_j^T$, the nearest source vertex lies outside the surface, so the signed depth and the loss term is zero. For target vertex $v_i^T$, the nearest source vertex is inside; the orthogonal projection onto the normal yields point $p$, and the signed depth $d_i>0$ produces a positive penalty.
• Figure 5: Visualization of phased optimization with scheduled weights. Rooster and comb across three phases. As the weights increase across phases, the search transitions from broad exploration to a focused zoom-in and local refinement. Phase-best results are marked with $\star$ and initialize the next phase.