Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction

TL;DR

This work tackles sketch-to-3D human pose estimation by introducing a learn-from-synthesis paradigm that uses diffusion priors to generate a large, diverse SKEP-120K dataset and a fast, end-to-end network. The model comprises a 2D Guidance Extractor, a Sketch Feature Extractor, and an SMPL Mesh Regressor, all conditioned through diffusion-based priors and trained with a quartet of losses that enforce joint-angle consistency, foreshortening alignment, SMPL pose fidelity, and shape regularization. Empirical results show state-of-the-art or competitive accuracy across multiple sketch styles while achieving orders-of-magnitude faster inference than prior optimization-based approaches, with strong generalization to unseen styles and even video sequences. Limitations include terminal-joint prediction in highly abstract sketches and the absence of subject-specific body-shape modeling, suggesting avenues for future extension.

Abstract

3D human pose estimation from sketches has broad applications in computer animation and film production. Unlike traditional human pose estimation, this task presents unique challenges due to the abstract and disproportionate nature of sketches. Previous sketch-to-pose methods, constrained by the lack of large-scale sketch-3D pose annotations, primarily relied on optimization with heuristic rules-an approach that is both time-consuming and limited in generalizability. To address these challenges, we propose a novel approach leveraging a "learn from synthesis" strategy. First, a diffusion model is trained to synthesize sketch images from 2D poses projected from 3D human poses, mimicking disproportionate human structures in sketches. This process enables the creation of a synthetic dataset, SKEP-120K, consisting of 120k accurate sketch-3D pose annotation pairs across various sketch styles. Building on this synthetic dataset, we introduce an end-to-end data-driven framework for estimating human poses and shapes from diverse sketch styles. Our framework combines existing 2D pose detectors and generative diffusion priors for sketch feature extraction with a feed-forward neural network for efficient 2D pose estimation. Multiple heuristic loss functions are incorporated to guarantee geometric coherence between the derived 3D poses and the detected 2D poses while preserving accurate self-contacts. Qualitative, quantitative, and subjective evaluations collectively show that our model substantially surpasses previous ones in both estimation accuracy and speed for sketch-to-pose tasks.

Figures (7)

• Figure 1: SKEP-120K Dataset Creating Pipeline. Three stages are involved: (I) generating diverse 3D poses (as SMPL); (II) adding random biases to bone lengths and projecting to 2D poses; (III) generating diverse text guidance; (IV) training a text-conditioned image generator for sketch synthesis.
• Figure 2: Data Description. SKEP-120K dataset comprises six sketch styles: cartoons, oil paintings, ink paintings, charcoal sketches, stick figures, and kids' drawings. The provided 2D/3D joints are shown on the left.
• Figure 3: Overall Pipeline. Given a sketch image as input, the network predicts 3D human poses represented by SMPL parameters. The overall network consists of three modules: a 2D guidance extractor as detailed in Sec. \ref{['sec: guidance']}; a sketch feature extractor as detailed in Sec. \ref{['sec: feature']}; and an SMPL regressor as detailed in Sec. \ref{['sec: regressor']}.
• Figure 4: Failure cases. Our model may predict inaccurate terminal joints.
• Figure 5: Qualitative Comparison of Multiple Sketch Styles. Our proposed model accurately predicts real human body proportions in cartoon images and outperforms other methods in various sketch styles. Its high performance across multiple sketch styles is attributed to our three-stage pose prediction network design and diverse dataset with perturbations. The red dashed box highlights the unreasonable 3D human pose estimation.