Diffusion for Natural Image Matting

TL;DR

This work tackles natural image matting by introducing DiffMatte, a diffusion-based iterative refinement that decouples the diffusion decoder from the matting encoder to reduce computational overhead while aligning training and inference noise sampling through Uniform Time Intervals (UTI). The approach uses a lightweight diffusion decoder that operates on top-level features from an arbitrary matting encoder and a mixed input of the image with its trimap to progressively refine the alpha matte via DDIM sampling. Key contributions include the decoupled, lightweight diffusion decoder (~5.3M parameters), the UTI self-aligned training strategy, and demonstrated state-of-the-art performance on Composition-1k with strong generalization to AIM-500 and other benchmarks. Practically, DiffMatte offers a flexible, efficient framework for diffusion-assisted matting that can integrate with existing matting architectures, enabling higher-quality mattes with lower overhead for interactive image editing.

Abstract

We aim to leverage diffusion to address the challenging image matting task. However, the presence of high computational overhead and the inconsistency of noise sampling between the training and inference processes pose significant obstacles to achieving this goal. In this paper, we present DiffMatte, a solution designed to effectively overcome these challenges. First, DiffMatte decouples the decoder from the intricately coupled matting network design, involving only one lightweight decoder in the iterations of the diffusion process. With such a strategy, DiffMatte mitigates the growth of computational overhead as the number of samples increases. Second, we employ a self-aligned training strategy with uniform time intervals, ensuring a consistent noise sampling between training and inference across the entire time domain. Our DiffMatte is designed with flexibility in mind and can seamlessly integrate into various modern matting architectures. Extensive experimental results demonstrate that DiffMatte not only reaches the state-of-the-art level on the Composition-1k test set, surpassing the best methods in the past by 5% and 15% in the SAD metric and MSE metric respectively, but also show stronger generalization ability in other benchmarks.

Figures (11)

• Figure 1: DiffMatte introduces the diffusion process to solve the natural image matting problem. By iteratively correcting the prediction, our method obtains state-of-the-art matting accuracy. DiffMatte can be embedded into arbitrary matting encoders, which makes its application scenarios more flexible and versatile
• Figure 2: Proposed DiffMatte Framework. DiffMatte provides noise sample $X$ for training through the forward diffusion process $q$, and uses X as an iterative prior to supplement the guidance of $c$ in the reverse process $p_\theta$. Diffusion decoder $\mathcal{D}$ serves as an iterative component and is independent of matting encoder$\mathcal{B}$, receiving matting features $F$ to predict the alpha matte at each step. As a universal decoder, it can cooperate with any matting encoder. $\mathcal{D}$ consists of Down-Block and Up-Block, which only contain convolutional layers and linear layers to process image information and time embedding $\tau$ respectively.
• Figure 3: Comparison of different training strategies. In contrast to the strategy proposed by ji2023ddp, we take into account the drift caused by prediction errors at each time step and propose a self-aligned strategy with uniform time intervals to align the noisy sample $X$ over all time steps.
• Figure 4: Qualitative results compared with previous SOTA methods on Composition-1k.
• Figure 5: The iteration results of each step of DiffMatte with different matting encoders.