Many-to-many Image Generation with Auto-regressive Diffusion Models

TL;DR

This work tackles the challenge of generating interrelated image sequences in broad contexts by introducing MIS, a large-scale dataset of 12M synthetic multi-image sets, and the Many-to-many Diffusion (M2M) framework. M2M uses an Image-Set Attention mechanism to autoregressively model arbitrary numbers of interconnected images, with two variants: M2M-Self and M2M-DINO, the latter leveraging external vision features. Trained on MIS, M2M learns to preserve content and style from preceding images and demonstrates strong zero-shot generalization to real images, while enabling task-specific fine-tuning for Novel View Synthesis and Visual Procedure Generation. The approach offers a scalable, domain-general solution for multi-image generation with practical impact on multi-view content, visual storytelling, and procedural generation in multimedia applications.

Abstract

Recent advancements in image generation have made significant progress, yet existing models present limitations in perceiving and generating an arbitrary number of interrelated images within a broad context. This limitation becomes increasingly critical as the demand for multi-image scenarios, such as multi-view images and visual narratives, grows with the expansion of multimedia platforms. This paper introduces a domain-general framework for many-to-many image generation, capable of producing interrelated image series from a given set of images, offering a scalable solution that obviates the need for task-specific solutions across different multi-image scenarios. To facilitate this, we present MIS, a novel large-scale multi-image dataset, containing 12M synthetic multi-image samples, each with 25 interconnected images. Utilizing Stable Diffusion with varied latent noises, our method produces a set of interconnected images from a single caption. Leveraging MIS, we learn M2M, an autoregressive model for many-to-many generation, where each image is modeled within a diffusion framework. Throughout training on the synthetic MIS, the model excels in capturing style and content from preceding images - synthetic or real - and generates novel images following the captured patterns. Furthermore, through task-specific fine-tuning, our model demonstrates its adaptability to various multi-image generation tasks, including Novel View Synthesis and Visual Procedure Generation.

Figures (17)

• Figure 1: Overview of Auto-regressive Diffusion Pipeline. During training, Auto-regressive Diffusion takes sets of noised latent images and their corresponding clean image features as inputs and then predicts the noise added to each noised latent image conditioned on the previous clean image features. At inference, our model can generate an arbitrary number of images in an auto-regressive manner by iteratively incorporating generated images back into the input.
• Figure 2: A sample image set of five distinctive images generated using a caption from Conceptual 12 M.
• Figure 3: Illustration of Image-Set Attention Module. The query token is denoted in a white square and its corresponding key/value attention region is marked by a diagonal striped pattern.
• Figure 4: Consistency Evaluation in M2M-Self and M2M-DINO: The figure showcases the ability of M2M-Self and M2M-DINO to maintain content (a) and style (b) consistency. Content consistency refers to the model's capacity to generate images with the same type of subject as preceding ones, while style consistency pertains to maintaining aesthetic elements like color schemes, textures, and artistic techniques. Each subfigure contains two panels: the top panel for M2M-Self and the bottom for M2M-DINO. Columns 1-4 showcase the preceding images for conditioning, and Columns 5-8 display images generated by the respective models.
• Figure 5: Generalization to Real Images. Columns 1-4 display the real images from the MSCOCO dataset, serving as the preceding images. Columns 5-8 showcase the corresponding images generated by M2M, conditioned on the preceding images.