VIP: Versatile Image Outpainting Empowered by Multimodal Large Language Model

TL;DR

VIP introduces a diffusion-based image outpainting framework that enables user-driven customization by converting image regions into space-aware text prompts using a Multimodal Large Language Model. A Center-Total-Surrounding (C-T-S) decoupled control mechanism aligns center and surrounding image regions with corresponding text tokens through separate cross-attention modules and adaptive fusion, enabling precise region-conditioned generation. The approach employs lightweight fine-tuning on an off-the-shelf Stable Diffusion model with two training regimes (SFT and GLT) and demonstrates state-of-the-art performance on Scenery, Building Facades, and WikiArt across unconditional and conditional settings, while supporting diverse customization such as irregular masks and keyword-driven content. This work offers a practical, efficient path to flexible, high-quality outpainting suitable for real-world applications where user guidance and region semantics matter, reducing training costs and enabling broader deployment.

Abstract

In this paper, we focus on resolving the problem of image outpainting, which aims to extrapolate the surrounding parts given the center contents of an image. Although recent works have achieved promising performance, the lack of versatility and customization hinders their practical applications in broader scenarios. Therefore, this work presents a novel image outpainting framework that is capable of customizing the results according to the requirement of users. First of all, we take advantage of a Multimodal Large Language Model (MLLM) that automatically extracts and organizes the corresponding textual descriptions of the masked and unmasked part of a given image. Accordingly, the obtained text prompts are introduced to endow our model with the capacity to customize the outpainting results. In addition, a special Cross-Attention module, namely Center-Total-Surrounding (CTS), is elaborately designed to enhance further the the interaction between specific space regions of the image and corresponding parts of the text prompts. Note that unlike most existing methods, our approach is very resource-efficient since it is just slightly fine-tuned on the off-the-shelf stable diffusion (SD) model rather than being trained from scratch. Finally, the experimental results on three commonly used datasets, i.e. Scenery, Building, and WikiArt, demonstrate our model significantly surpasses the SoTA methods. Moreover, versatile outpainting results are listed to show its customized ability.

Figures (15)

• Figure 1: The visualization of versatile outpainting results generated by our proposed method. Unlike existing methods (left), our method (right) can generate satisfactory outpainting results in both unconditional and conditional paradigms.
• Figure 2: Overall framework of our proposed learning method. First, we concatenate the raw image latent, masked image latent, and mask as the input of the unet. The output will be calculated as mean-square error (MSE) loss with the added noise in the raw image latent. Meanwhile, the text prompt and mask will be input into the proposed C-T-S decoupled control mechanism as conditions. 'R' and 'C' mean the resize and concatenation operations, respectively.
• Figure 3: Architecture of Center-Total-Surrounding decoupled control mechanism. '$\cdot$', '$\times$', and '+' mean the element-wise multiplication, tensor multiplication and element-wise addition, respectively.
• Figure 4: Visualization comparison bewteen different methods on the Scenery dataset. VIP can surpass other SoTA methods in the concerned box region.
• Figure 5: Visualization comparison under different outpainting mask types.