AsyncDSB: Schedule-Asynchronous Diffusion Schrödinger Bridge for Image Inpainting

TL;DR

This work identifies a fundamental misalignment in diffusion Schrödinger bridge-based image inpainting: applying a single, pixel-synchronous noise schedule β_t fails to match the true, asynchronous restoration dynamics across image frequencies. It introduces AsyncDSB, a two-step framework that first predicts the missing image gradients and then applies a gradient-guided, per-pixel asynchronous diffusion process, controlled by per-pixel τ_{i,j} schedules. Experiments on CelebA-HQ and Places2 show that AsyncDSB consistently improves FID by approximately 3%–14% over the I2SB baseline, with notable gains on center/half masks and richer textures in generated outputs. The approach demonstrates how incorporating frequency priors and per-pixel scheduling can materially enhance diffusion-bridge-based inpainting and opens avenues for more flexible, frequency-aware diffusion strategies in conditional generation.

Abstract

Image inpainting is an important image generation task, which aims to restore corrupted image from partial visible area. Recently, diffusion Schrödinger bridge methods effectively tackle this task by modeling the translation between corrupted and target images as a diffusion Schrödinger bridge process along a noising schedule path. Although these methods have shown superior performance, in this paper, we find that 1) existing methods suffer from a schedule-restoration mismatching issue, i.e., the theoretical schedule and practical restoration processes usually exist a large discrepancy, which theoretically results in the schedule not fully leveraged for restoring images; and 2) the key reason causing such issue is that the restoration process of all pixels are actually asynchronous but existing methods set a synchronous noise schedule to them, i.e., all pixels shares the same noise schedule. To this end, we propose a schedule-Asynchronous Diffusion Schrödinger Bridge (AsyncDSB) for image inpainting. Our insight is preferentially scheduling pixels with high frequency (i.e., large gradients) and then low frequency (i.e., small gradients). Based on this insight, given a corrupted image, we first train a network to predict its gradient map in corrupted area. Then, we regard the predicted image gradient as prior and design a simple yet effective pixel-asynchronous noise schedule strategy to enhance the diffusion Schrödinger bridge. Thanks to the asynchronous schedule at pixels, the temporal interdependence of restoration process between pixels can be fully characterized for high-quality image inpainting. Experiments on real-world datasets show that our AsyncDSB achieves superior performance, especially on FID with around 3% - 14% improvement over state-of-the-art baseline methods.

Figures (8)

• Figure 1: Connection between noise schedule ($\beta_t$) and the restoration process of I$^2$SBliu2023i2sb.
• Figure 2: Qualitative Analysis.
• Figure 3: Comparison between theoretical schedule and practical process on different frequency.
• Figure 4: The overall framework of our AsyncDSB, which consists of two steps, an image gradient completion step, and a gradient-guided asynchronous Schrödinger diffusion bridge step. Here, given a corrupted image, the Step 1 aims to predict the image gradient of the missing area. Then, the Step 2 accounts for assigning a pixel-asynchronous noise schedule, which is employed to make a asynchronous Schrödinger diffusion bridge for completing the corrupted region in origin pixel space according to the image gradient information obtained by the Step 1.
• Figure 5: Qualitative results on CelebA-HQ and Places2 under regular and irregular masks.