StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback

TL;DR

This work targets the challenge of generating pixel-based, hand-drawn sketches with diffusion models. It introduces StableSketcher, which (i) fine-tunes the VAE for improved latent reconstruction and (ii) employs a VQA-guided DDPO framework to maximize prompt fidelity, using a novel VQA reward that leverages SketchDUO’s instance-level QA signals. A first-of-its-kind SketchDUO dataset provides 35.8K positive/negative sketches with fine-grained captions and 54.37K QA triplets to support instance-level sketch learning and contrastive modeling. Empirical results show StableSketcher achieves superior FID and TIFAScore compared to Stable Diffusion baselines, with user studies confirming improved sketch quality and alignment to prompts. The work advances practical abstract sketch generation and offers a dataset and methodology for fine-grained multimodal control in diffusion-based sketch synthesis, with potential for broader sketch-guided creative applications.

Abstract

Although recent advancements in diffusion models have significantly enriched the quality of generated images, challenges remain in synthesizing pixel-based human-drawn sketches, a representative example of abstract expression. To combat these challenges, we propose StableSketcher, a novel framework that empowers diffusion models to generate hand-drawn sketches with high prompt fidelity. Within this framework, we fine-tune the variational autoencoder to optimize latent decoding, enabling it to better capture the characteristics of sketches. In parallel, we integrate a new reward function for reinforcement learning based on visual question answering, which improves text-image alignment and semantic consistency. Extensive experiments demonstrate that StableSketcher generates sketches with improved stylistic fidelity, achieving better alignment with prompts compared to the Stable Diffusion baseline. Additionally, we introduce SketchDUO, to the best of our knowledge, the first dataset comprising instance-level sketches paired with captions and question-answer pairs, thereby addressing the limitations of existing datasets that rely on image-label pairs. Our code and dataset will be made publicly available upon acceptance.

Figures (14)

• Figure 1: An overview of our StableSketcher framework and SketchDUO dataset.
• Figure 2: (a) Proportional distribution of the six categories in SketchDUO, shown as the number of categories and their respective percentages. (b) Number of data samples within each category in SketchDUO, displayed as counts and their respective percentages. (c) Class-level percentage distribution in the positive dataset. (d) Class-level percentage distribution in the negative dataset.
• Figure 3: Background-preserving sketch augmentations: original; rotation by $-15^\circ$ with white padding; rotation by $+15^\circ$ with white padding; Gaussian blur (weak, $k=3$, $\sigma=0.8$); Gaussian noise applied to strokes only (weak, $\sigma=8$); Gaussian blur (strong, $k=5$, $\sigma=1.6$); Gaussian noise applied to strokes only (strong, $\sigma=16$); and line thickening obtained by binarization followed by morphological dilation with a $3\times3$ elliptical kernel for one iteration.
• Figure 4: Overall architecture of StableSketcher. The input prompt ($T$) is fed into the diffusion model through the CLIP text encoder, where the VAE is fine-tuned using $\mathcal{L}_{\text{recon}}$ and $\mathcal{L}_{\text{LPIPS}}$. Once the diffusion model generates the image ($\hat{I}$), it is passed to the fine-tuned mPLUG-large, along with the question from SketchDUO, to generate the corresponding answer. The VQA-based reward $\mathcal{R}_{\text{VQA}}$, calculated using the TIFAScore, is then used as a feedback signal in the reinforcement learning process.
• Figure 5: Effect of KL weight on VAE reconstruction quality. In the reconstructed images, $\alpha$ denotes the coefficient in the VAE loss, $\mathrm{L_{recon}} + \alpha\cdot\mathrm{L_{KL}}$.