Empowering LLMs to Understand and Generate Complex Vector Graphics

TL;DR

The paper introduces LLM4SVG, a framework that enables large language models to understand and generate complex SVG graphics. It tackles semantic alignment and rendering order by introducing 55 SVG semantic tokens and a two-stage instruction-tuning pipeline, leveraging a new SVGX-SFT dataset with $250{,}000$ SVGs and $580{,}000$ instruction-following samples. Through comprehensive experiments, LLM4SVG achieves superior performance among LLM-based SVG methods, offering precise numeric coordinates and hex color support while delivering faster generation than optimization-based approaches. The approach bridges natural language processing and vector graphics, with potential impact on UI design, data visualization, and educational tools. The work also outlines future directions for cross-modal editing, domain adaptation, and iterative natural-language-driven vector editing.

Abstract

The unprecedented advancements in Large Language Models (LLMs) have profoundly impacted natural language processing but have yet to fully embrace the realm of scalable vector graphics (SVG) generation. While LLMs encode partial knowledge of SVG data from web pages during training, recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions. Additionally, LLM training typically lacks modeling and understanding of the rendering sequence of vector paths, which can lead to occlusion between output vector primitives. In this paper, we present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics. LLM4SVG facilitates a deeper understanding of SVG components through learnable semantic tokens, which precisely encode these tokens and their corresponding properties to generate semantically aligned SVG outputs. Using a series of learnable semantic tokens, a structured dataset for instruction following is developed to support comprehension and generation across two primary tasks. Our method introduces a modular architecture to existing large language models, integrating semantic tags, vector instruction encoders, fine-tuned commands, and powerful LLMs to tightly combine geometric, appearance, and language information. To overcome the scarcity of SVG-text instruction data, we developed an automated data generation pipeline that collected our SVGX-SFT Dataset, consisting of high-quality human-designed SVGs and 580k SVG instruction following data specifically crafted for LLM training, which facilitated the adoption of the supervised fine-tuning strategy popular in LLM development.

Figures (13)

• Figure 1: Our LLM4SVG can understand and generate vector graphics from textual description. Our LLM4SVG is designed to: (a) Understand the semantics of SVG (Scalable Vector Graphics) source code and directly extract the meanings conveyed by vector images; (b) Generate corresponding structured SVG representations from textual prompts and decode them into SVG source code that accurately reflects the described content. (c) illustrates some SVG examples generated by our method.
• Figure 2: Overview of SVGX-SFT Dataset.(a)Top Frequent Words in SVGX-SFT Dataset. The most frequently occurring words in the SVGX-SFT dataset, highlighting common patterns and terminology used in SVG metadata. (b)Comparison of SVG Element Counts Before and After Cleaning. A comparison of the number of SVG elements before and after the cleaning process. The reduction in element count demonstrates the effectiveness of our preprocessing steps. (c)Word Cloud of SVG Names and Descriptions. A word cloud visualization of SVG names and descriptions, illustrating the distribution and emphasis of different terms in the dataset. (d)Group (<g></g>) Number Distribution. The distribution of <g> (group) elements in the dataset, showing the frequency of grouped elements and their structural significance within SVG files.
• Figure 3: An Overview of LLM4SVG. Our LLM4SVG is capable of understanding and generating SVGs effectively. (1) During the training phase, we provide both the original SVG code $\mathbf{X}_{v}$ and the corresponding instruction data $\mathbf{X}_{\mathrm{inst}}$ as input. For the understanding task, we use detailed descriptions $\mathbf{X}_{a}$ generated by GPT-4 GPT4 as the training labels. For the generation task, the SVG code portion is masked and serves as the target that the model needs to predict. (2) During the inference phase, for the understanding task, given an SVG source code, the model generates a description that aligns with the semantics expressed by the SVG. For the generation task, the model generates an SVG based on the input text prompt. During both training and inference phases, the rendered image $\mathbf{X}_{\mathrm{img}}$ of the SVG can be used as conditional input to the model, guiding the content that the model understands or generates.
• Figure 4: Qualitative comparison of LLM4SVG with state-of-the-art SVG generation methods, categorized into optimization-based and LLM-based approaches. The prompts (left) guide the generation of vector graphics across different methods. Optimization-based methods (left section) focus on refining SVGs through iterative optimization, while LLM-based methods (right section) generate SVGs directly from text descriptions. The results highlight differences in abstraction, structure, and fidelity to the input prompt, with our LLM4SVG achieving more structured and visually coherent outputs.
• Figure S1: Visual Comparison between Decimal Coordinates and Integer Coordinates in SVGs. Only integer coordinates will lead to shape distortions and incompletion.