MROSS: Multi-Round Region-based Optimization for Scene Sketching

TL;DR

MROSS addresses the challenge of converting a scene into a concise, vector-based sketch by introducing region-based, multi-round optimization that builds sketches from Bézier curves. The method allocates and samples strokes per region, guided by edge information, and iteratively refines the sketch from a global region to localized regions. It leverages CLIP-based Semantic Loss and a VGG-based Feature Loss to balance semantic content and structural fidelity, yielding adjustable abstraction levels validated by quantitative metrics (LPIPS, SSIM) and user studies. The approach demonstrates competitive performance against state-of-the-art methods and offers a flexible pipeline suitable for design and animation workflows where region emphasis and vector representations are important.

Abstract

Scene sketching is to convert a scene into a simplified, abstract representation that captures the essential elements and composition of the original scene. It requires a semantic understanding of the scene and consideration of different regions within the scene. Since scenes often contain diverse visual information across various regions, such as foreground objects, background elements, and spatial divisions, dealing with these different regions poses unique difficulties. In this paper, we define a sketch as some sets of Bézier curves because of their smooth and versatile characteristics. We optimize different regions of input scene in multiple rounds. In each optimization round, the strokes sampled from the next region can seamlessly be integrated into the sketch generated in the previous optimization round. We propose an additional stroke initialization method to ensure the integrity of the scene and the convergence of optimization. A novel CLIP-based Semantic Loss and a VGG-based Feature Loss are utilized to guide our multi-round optimization. Extensive experimental results on the quality and quantity of the generated sketches confirm the effectiveness of our method.

Figures (8)

• Figure 1: Drawings of different scenes by different artists. Notice the significant differences in level of abstraction between different regions of the drawings.
• Figure 2: Method overview – Given a scene photograph, and the selected regions, stroke initialization (stroke allocation and stroke sampling) is used to determine the initial strokes locations (red points) of different regions. These initial stroke locations will be converted into Bézier curves (black curves) as input of our multi-round optimization. In the bottom we show the details of the loss function during optimization.
• Figure 3: An illustration of FPS process, which ensures that the sample points are well-spaced in the edge image.
• Figure 4: The process of the optimization. The optimization results of the previous round will be superimposed with the initial strokes of the current region as the optimization input for the next round.
• Figure 5: Different Stroke Sampling Method. Top to Bottom: FPS sampling method, the sampling method of CLIPasso vinker2022clipasso, random sampling.