LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model

TL;DR

LineArt tackles the challenge of rendering photorealistic appearance onto professional line drawings without requiring 3D modeling or training. It combines a knowledge-guided, training-free diffusion framework with a two-stage appearance transfer inspired by Imprimatura: Base Layer Shaping to model lighting and Surface Layer Coloring to encode texture, using a multi-frequency line fusion to preserve structure. The method utilizes ControlNet and IP-Adapter to guide generation and introduces the ProLines dataset of 5101 professional line drawings for evaluation. Experiments show LineArt achieves superior accuracy, realism, and material fidelity compared with state-of-the-art baselines, validated by quantitative metrics and user studies. This approach offers designers a practical, interpretable, zero-shot tool for high-quality design drawing appearance transfer.

Abstract

Image rendering from line drawings is vital in design and image generation technologies reduce costs, yet professional line drawings demand preserving complex details. Text prompts struggle with accuracy, and image translation struggles with consistency and fine-grained control. We present LineArt, a framework that transfers complex appearance onto detailed design drawings, facilitating design and artistic creation. It generates high-fidelity appearance while preserving structural accuracy by simulating hierarchical visual cognition and integrating human artistic experience to guide the diffusion process. LineArt overcomes the limitations of current methods in terms of difficulty in fine-grained control and style degradation in design drawings. It requires no precise 3D modeling, physical property specs, or network training, making it more convenient for design tasks. LineArt consists of two stages: a multi-frequency lines fusion module to supplement the input design drawing with detailed structural information and a two-part painting process for Base Layer Shaping and Surface Layer Coloring. We also present a new design drawing dataset ProLines for evaluation. The experiments show that LineArt performs better in accuracy, realism, and material precision compared to SOTAs.

Figures (10)

• Figure 1: Generated results by our method and Comparison with SOTAs. (a): Given an object-centered design drawing and a photo as the reference, our method transfers the complex appearance features in the photo to the fine structure of the design drawing with high fidelity. (b): Compared with ZeST ZeST, StyleID StyleID, AesPA-Net AesPA-Net, Cross-Image Attention crossimageattention, and T2I-Adapter-SDXL T2I-Adapter. Our method better ensures the accurate transfer of complex appearance features and maintains the fine structure of the original design drawing.
• Figure 2: (a) Various pencils in hardness and color. (b) Le Taureau by Picasso depicts a shift from realism to abstraction. ©"Pasadena, Norton Simon Museum, Picasso P. The Bull, 1946" photo by Vahe Martirosyan, [CC BY-SA 2.0] via https://bit.ly/3MFB3pm. (c) Visual representations of Le Taureau.
• Figure 3: A classical painting technique: Imprimatura. We designate the Underpainting process as Base Layer Shaping and Glazing as Surface Layer Coloring. In the Base Layer Shaping stage, we handle implicit information from the reference image, such as lighting effects, illumination, and shading-based reflectance. Features related to texel and color are addressed in the Surface Layer Coloring stage.
• Figure 4: Our Workflow. The process begins with a design drawing $L_{\text{original}}$ and an appearance image $I_{\text{appearance}}$. Depth-based ControlNet estimates depth and generates soft edges to guide the synthesis. (a) The Multi-frequency Line Fusion module employs assertion-guided techniques to enhance structural detail control. (b) Base Layer Shaping decomposes the illumination of the appearance image using a multi-scale retinex approach, generating retinex illumination layers $L_{\text{retinex}}$ to balance brightness. (c) Surface Layer Coloring refines the output by utilizing layout and style blocks in a U-net with cross-attention for accurate material embedding.
• Figure 5: Moving Background's Impact on Texture Synthesis. By removing irrelevant background regions, materials embedding become more accurate materials embedding.