Text-to-Vector Generation with Neural Path Representation

TL;DR

This work addresses the challenge of producing high-quality, edit-friendly SVGs from text prompts by introducing a neural path representation. It learns a shared latent space for vector path data using a dual-branch VAE trained on both control-point sequences and rendered path images, ensuring geometric validity and rendering fidelity. A two-stage text-driven optimization then guides SVG generation: Stage 1 uses Variational Score Distillation with a diffusion prior to align with the prompt, and Stage 2 applies layer-wise vectorization with path simplification and CLIP/IoU-guided refinement to enforce a clear, hierarchical structure. The approach yields SVGs with smooth, well-organized paths and strong alignment to prompts, demonstrated via vector-, image-, and text-level metrics, user studies, and ablations, and enables applications in adjustable detail, styling, customization, image-to-SVG, and animation. Although slower than raster-to-vector pipelines, the method lays groundwork for fast feed-forward T2V models and broader vector-based design tools.

Abstract

Vector graphics are widely used in digital art and highly favored by designers due to their scalability and layer-wise properties. However, the process of creating and editing vector graphics requires creativity and design expertise, making it a time-consuming task. Recent advancements in text-to-vector (T2V) generation have aimed to make this process more accessible. However, existing T2V methods directly optimize control points of vector graphics paths, often resulting in intersecting or jagged paths due to the lack of geometry constraints. To overcome these limitations, we propose a novel neural path representation by designing a dual-branch Variational Autoencoder (VAE) that learns the path latent space from both sequence and image modalities. By optimizing the combination of neural paths, we can incorporate geometric constraints while preserving expressivity in generated SVGs. Furthermore, we introduce a two-stage path optimization method to improve the visual and topological quality of generated SVGs. In the first stage, a pre-trained text-to-image diffusion model guides the initial generation of complex vector graphics through the Variational Score Distillation (VSD) process. In the second stage, we refine the graphics using a layer-wise image vectorization strategy to achieve clearer elements and structure. We demonstrate the effectiveness of our method through extensive experiments and showcase various applications. The project page is https://intchous.github.io/T2V-NPR.

Figures (16)

• Figure 1: Our system pipeline starts with learning a neural representation of paths by training a dual-branch VAE. Next, we optimize the SVG, represented with neural paths, to align with the provided text prompt, which is achieved through a two-stage path optimization process.
• Figure 2: Training loss of the dual-branch VAE.
• Figure 3: Path samples and latent interpolations. (a) Path examples from the dataset. (b) Path samples decoded from random vectors in the latent space. (c) Latent interpolations among four given samples (marked in red).
• Figure 4: Layer-wise optimization strategy.
• Figure 5: Qualitative comparison to vectorization with T2I methods.