Layered Image Vectorization via Semantic Simplification

TL;DR

The paper tackles the challenge of vectorizing raster images into scalable, semantically structured layers by introducing progressive SDS-based image simplification to guide a two-stage vector reconstruction. The method first builds structure-wise vectors aligned to semantic masks through a layer-aware loss, then refines visuals by freezing structure vectors and optimizing color and additional primitives via a visual fidelity objective. Key contributions include the SDS-driven progressive abstraction, layer-wise structure loss with dedicated overlap handling, and a VeC-based assessment of layer compactness, all leading to improved visual fidelity and semantic alignment with a compact layered representation. This approach enhances editing usability and semantic interpretation across diverse image types, outperforming several state-of-the-art vectorization methods in both fidelity and interpretability.

Abstract

This work presents a progressive image vectorization technique that reconstructs the raster image as layer-wise vectors from semantic-aligned macro structures to finer details. Our approach introduces a new image simplification method leveraging the feature-average effect in the Score Distillation Sampling mechanism, achieving effective visual abstraction from the detailed to coarse. Guided by the sequence of progressive simplified images, we propose a two-stage vectorization process of structural buildup and visual refinement, constructing the vectors in an organized and manageable manner. The resulting vectors are layered and well-aligned with the target image's explicit and implicit semantic structures. Our method demonstrates high performance across a wide range of images. Comparative analysis with existing vectorization methods highlights our technique's superiority in creating vectors with high visual fidelity, and more importantly, achieving higher semantic alignment and more compact layered representation. The project homepage is https://szuviz.github.io/layered_vectorization/.

Figures (25)

• Figure 1: Layered vectorization: by generating a sequence of progressive simplified images (top row), our technique reconstructs vectors layer by layer, from macro to finer details (middle row). Our approach maintains the vectors compactly aligned within the boundaries of explicit and implicit semantic objects (bottom row).
• Figure 2: Vectors with levels of detail generated with our method: from left to right, vector primitives from macro to finer details are added layer by layer.
• Figure 3: Layered vectorization pipeline: with the input of a target image, its sequence of progressive simplified images is generated using the SDS diffusion model. Vectors are then reconstructed in two stages: structure construction via layer-wise shape optimization to match segmented masks and visual refinement for high fidelity.
• Figure 4: Example of SDS-based image simplification: (a) a sequence of progressively simplified images, with the original image (level 0) on the left; (b) comparison of segmented masks between levels 0 and 4, showing that more simplified image captures more macro structures, such as the 'whole body of the robot' detected in level 4 but not in level 0.
• Figure 5: Example of the feature-averaging effect in SDS: as optimization progresses, the 'robot' loses fine details, such as 'fingers' and 'helmet', showing the smoothed and simplified appearance.