Image-Space Collage and Packing with Differentiable Rendering

TL;DR

The paper shifts 2D collage and packing from object-space optimization to image-space optimization using differentiable rendering, enabling gradient-based fitting of diverse shapes without task-specific geometric descriptors. By representing each item with $N$ cubic Bézier curves and optimizing translations $\mathbf{t}$, rotations $\mathbf{r}$, and scales $\mathbf{s}$ to minimize a composite loss $\mathcal{L}=\alpha\mathcal{L}_{containment}+\beta\mathcal{L}_{overlap}+\gamma\mathcal{L}_{uniform}$, the method achieves robust containment, minimal overlap, and uniform distribution. A hierarchical resolution strategy—from $50\times50$ up to $600\times600$—drives fast convergence while preserving accuracy. Empirical results show order-of-magnitude speedups over baselines and demonstrate versatility across icons, text, and data-visualization collages, with ablations validating the contributions of uniform loss and multiscale optimization. The work suggests promising extensions to interactive and text-driven collage methods and potential hybridization with object-space losses for enhanced control and quality.

Abstract

Collage and packing techniques are widely used to organize geometric shapes into cohesive visual representations, facilitating the representation of visual features holistically, as seen in image collages and word clouds. Traditional methods often rely on object-space optimization, requiring intricate geometric descriptors and energy functions to handle complex shapes. In this paper, we introduce a versatile image-space collage technique. Leveraging a differentiable renderer, our method effectively optimizes the object layout with image-space losses, bringing the benefit of fixed complexity and easy accommodation of various shapes. Applying a hierarchical resolution strategy in image space, our method efficiently optimizes the collage with fast convergence, large coarse steps first and then small precise steps. The diverse visual expressiveness of our approach is demonstrated through various examples. Experimental results show that our method achieves an order-of-magnitude speedup compared to state-of-the-art techniques. The project page is https://szuviz.github.io/pixel-space-collage-technique/.

Figures (17)

• Figure 1: Image-space collage and packing framework: starting with initialized 2D geometric items and their transformations, image-space losses are computed between the rasterized image and the target shape of the collage container across a hierarchy of image resolutions. These losses are then used to iteratively update the transformation parameters, refining the arrangement of the geometric items.
• Figure 2: MAT-based position initialization: with the detected medial axes and their nearest associated widths to the boundary (left), visual elements are initialized in the way that larger ones are placed on axes with larger medial widths (right).
• Figure 3: Trade-off between collage quality and computation time for different image resolutions. Note that the three collages on the right have been resized for better visualization of quality differences and do not reflect their original resolution.
• Figure 4: Packing examples with attracting forces: (left) our technique integrates a centripetal or downward force to pack elements efficiently within an open area; (right) using a collage container, elements are first attracted and confined within specific shapes.
• Figure 5: Gradual optimization creates animation effects: (top) an expanding animation effect, (bottom) a falling down animation effect.