Sketch-guided Image Inpainting with Partial Discrete Diffusion Process

TL;DR

This paper introduces sketch-guided image inpainting using a Partial Discrete Diffusion Process (PDDP) that operates in a learned discrete latent space. It couples a forward diffusion on masked latent tokens with a sketch-conditioned reverse diffusion implemented by a bidirectional transformer, enabling control over object shape and pose via hand-drawn sketches. A two-stage pipeline first learns a discrete latent codebook and then performs partial diffusion on masked tokens, with a sketch encoder supplying conditioning information; a MS-COCO-derived dataset with object sketches is synthesized for training and evaluation. Across quantitative metrics (FID/LPIPS with Local variants) and user studies, the method achieves state-of-the-art results, demonstrating strong fidelity to sketches and context alignment, and is made reproducible through publicly available code.

Abstract

In this work, we study the task of sketch-guided image inpainting. Unlike the well-explored natural language-guided image inpainting, which excels in capturing semantic details, the relatively less-studied sketch-guided inpainting offers greater user control in specifying the object's shape and pose to be inpainted. As one of the early solutions to this task, we introduce a novel partial discrete diffusion process (PDDP). The forward pass of the PDDP corrupts the masked regions of the image and the backward pass reconstructs these masked regions conditioned on hand-drawn sketches using our proposed sketch-guided bi-directional transformer. The proposed novel transformer module accepts two inputs -- the image containing the masked region to be inpainted and the query sketch to model the reverse diffusion process. This strategy effectively addresses the domain gap between sketches and natural images, thereby, enhancing the quality of inpainting results. In the absence of a large-scale dataset specific to this task, we synthesize a dataset from the MS-COCO to train and extensively evaluate our proposed framework against various competent approaches in the literature. The qualitative and quantitative results and user studies establish that the proposed method inpaints realistic objects that fit the context in terms of the visual appearance of the provided sketch. To aid further research, we have made our code publicly available at https://github.com/vl2g/Sketch-Inpainting .

Figures (6)

• Figure 1: Sketch-guided Image Inpainting has been an under-explored task in the literature and is often restricted to partial sketch-based image manipulation jo2019scdeepfillv2deepfillv2. We fill this gap in the literature by proposing a novel partial discrete diffusion process for sketch-guided object-level inpainting. Our proposed approach significantly outperforms other plausible approaches on Sketch-guided Image Inpainting.
• Figure 1: Evaluation of our model with non-synthetic sketches with varying styles and levels of detail. Sketches are sourced from the web.
• Figure 2: Our method involves obtaining a discrete latent space representation of the original image and its masked counterpart using a pretrained VQ-VAE. The image is first converted to noise by iteratively adding noise to the masked region in the forward process of the proposed Partial Discrete Diffusion Process (Section \ref{['subsec:discrete_diff']}). In the reverse process, sketch-guided inpainting is performed iteratively using a sketch-guided bi-directional transformer model that takes the masked image tokens and the query sketch. It predicts the tokens of the missing regions (Section \ref{['subsubsec:reverse_process']}). By iteratively refining the inpainted image using the sketch and the available information from the original image, the proposed method can generate high-quality inpainted images with correct visual and pose details. (Best viewed in color).
• Figure 2: Additional results for the qualitative comparison of the proposed method with competing baselines.
• Figure 3: An example of our dataset. We randomly mask a bounding box shown using red color and provide the masked image along with the corresponding sketch as input to the image inpainting method. Please refer to Section \ref{['subsec:dataset']} for more details.