PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference

TL;DR

This research provides a framework for incorporating human preference into the iterative refinement of generative models based on modeling reward accuracy, with broad implications for the design of visually driven AI applications.

Abstract

In this paper, we make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework, significantly improving the quality and visual appeal of inpainted images. Specifically, instead of directly measuring the divergence with paired images, we train a reward model with the dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Then, we adopt a reinforcement learning process to fine-tune the distribution of a pre-trained diffusion model for image inpainting in the direction of higher reward. Moreover, we theoretically deduce the upper bound on the error of the reward model, which illustrates the potential confidence of reward estimation throughout the reinforcement alignment process, thereby facilitating accurate regularization. Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach, showing significant improvements in the alignment of inpainted images with human preference compared with state-of-the-art methods. This research not only advances the field of image inpainting but also provides a framework for incorporating human preference into the iterative refinement of generative models based on modeling reward accuracy, with broad implications for the design of visually driven AI applications. Our code and dataset are publicly available at https://prefpaint.github.io.

Figures (22)

• Figure 1: Visual comparisons of the results by the diffusion-based image inpainting model named "Runway", and the aligned model through the proposed method.
• Figure 2: Experimental plot of reward prediction error vs.$\| z\|_{\mathbf{V}^{-1}}$ on the validation set, where a dashed line is an upper boundary of error, positively relative to $\| z\|_{\mathbf{V}^{-1}}$.
• Figure 2: Comparison across metrics: higher values are better for all metrics except "Rank".
• Figure 3: Statistical characteristics of the dataset we constructed. (a) the score distribution of the images across different selected datasets; (b) the comparison between the distribution of the average score and score for details; (c) and (d) show the numbers of images with different mask ratios on the outpainting and warping splits, respectively.
• Figure 3: Left: Ablation studies on amplification factors, where "static" refers to employing a constant factor to replace $\gamma$ in Eq. \ref{['eq:out']}. The column "Factor" indicates an average magnitude of amplification strength, i.e., $\mathbb{E}_{\mathbf{z}\sim \mathbf{p}(\mathbf{z})}(\gamma)$. For our method, we coordinate the value of $k$ in Eq. \ref{['Eq:Scaling']} to change the $\mathbb{E}(\gamma)$ shown in the Table. "Acl." signifies acceleration, calculated by $\frac{T_{b}}{T_{m}} - 1$ with $T_{b}$ and $T_{m}$ being the convergence iterations of baseline and compared methods, respectively. For all metrics, the larger, the better. Right: Performance of the reward model trained with two manners based on pre-trained CLIP clip with various fix rates (FRs). "Acc" and "Var" stand for the accuracy and variance of the reward estimation, respectively. "Bd." is the ratio of data below the same upper boundary. (The underlined settings are selected.)