SDGraph: Multi-Level Sketch Representation Learning by Sparse-Dense Graph Architecture

TL;DR

SDGraph tackles the challenge of learning from sparse, abstract freehand sketches by introducing a Multi-Level Sketch Representation Scheme and a novel two-graph architecture. The Sparse Graph captures sketch- and stroke-level information, while the Dense Graph learns sketch- and point-level information, with an information fusion bridge enabling cross-graph collaboration; a preprocess stage standardizes inputs. Across classification, zero-shot sketch-based image retrieval, and generation tasks, SDGraph consistently surpasses state-of-the-art methods and demonstrates robust cross-task generalization. The work provides a principled view of which sketch information is effective and how to leverage it through multi-level graph representations, with practical impact for sketch-based recognition, retrieval, and synthesis.

Abstract

Freehand sketches exhibit unique sparsity and abstraction, necessitating learning pipelines distinct from those designed for images. For sketch learning methods, the central objective is to fully exploit the effective information embedded in sketches. However, there is limited research on what constitutes effective sketch information, which in turn constrains the performance of existing approaches. To tackle this issue, we first proposed the Multi-Level Sketch Representation Scheme to systematically identify the effective information. The scheme organizes sketch representation into three levels: sketch-level, stroke-level, and point-level. This design is based on the granularity of analytical elements, from coarse (sketch-level) to fine (point-level), thereby ensuring more comprehensive coverage of the sketch information. For each level, we conducted theoretical analyses and experimental evaluations to identify and validate the effective information. Building on the above studies, we developed SDGraph, a deep learning architecture designed to exploit the identified effective information across the three levels. SDGraph comprises two complementary modules: a Sparse Graph that treats strokes as nodes for sketch-level and stroke-level representation learning, and a Dense Graph that treats points as nodes for sketch-level and point-level representation learning. Both modules employ graph convolution along with down-sampling and up-sampling operations, enabling them to function as both encoder and decoder. Besides that, an information fusion module bridges the two graphs to further enhance feature extraction. SDGraph supports a wide range of sketch-related downstream tasks, achieving accuracy improvements of 1.15\% and 1.70\% over the state-of-the-art in classification and retrieval, respectively, and 36.58\% improvement in vector sketch generation quality.

Figures (13)

• Figure 1: Multi-Level Sketch Representation Scheme. The not considered information is not beneficial for the downstream tasks covered in this study, and is therefore not considered in our framework. However, we acknowledge that they may be beneficial in other contexts or applications.
• Figure 2: Overall workflow. The input free-hand sketch is first processed by the preprocess module, which filters out outliers, applies translation, scaling, and resampling to normalize the sketch. Subsequently, the init graph procedure constructs the initial SGraph and DGraph (S&D Graphs) from the preprocessed sketch. The S&D Graphs are then fed into the SD Encoder, which updates node features and applies down-sampling operations to accelerate feature extraction. After passing through multiple SD Encoders, the model branches based on the downstream task: for classification or retrieval tasks, the resulting S&D Graphs are processed by the SD Pooling and Multi-Layer Perceptions (MLPs) to generate the global feature. For generation tasks, the SD Decoder updates the node features and performs up-sampling to restore the S&D Graphs to their original scale. Finally, SGraph features are transferred to the DGraph to produce point-wise features.
• Figure 3: Stroke encoding module. The points of each individual stroke are processed by the 1D convolution modules, and the resulting features are aggregated via max pooling to obtain the stroke representation.
• Figure 4: Sparse Graph down-sample.Top side: Using FPS to select a set of representative SGraph center nodes, where the initial SGraph node features serve as the stroke coordinates. Subsequently, KNN is employed to identify the neighboring stroke nodes. Finally, the neighbor fusion process (N-Fusion) operation is performed to aggregate the neighbor node features, as shown in \ref{['nfusion']}. Bottom side: The FPS and KNN results computed in the SGraph are propagated to the DGraph, and the DGraph undergoes corresponding down-sampling and N-Fusion operations.
• Figure 5: N-Fusion process.Top side: For the SGraph nodes, the features of the center nodes (brown) are concatenated with those of the neighboring nodes (green), and the concatenated feature is fed into MLPs for further processing. Bottom side: For the DGraph nodes, the feature-space distance (Distance 1) is first computed using the original order of the neighboring nodes. The neighbor node sequence is then reversed, and a second distance (Distance 2) is calculated. The smaller of the two distances is used to establish the optimal node correspondence. Finally, the corresponding features are concatenated and updated via a GCN.