ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction

TL;DR

ArtAug tackles the challenge of aligning diffusion-based text-to-image models with human aesthetics by introducing a synthesis-understanding interaction framework. An understanding multimodal LLM analyzes generated images and supplies fine-grained, localized modification prompts, which are applied in a partitioned fashion and distilled into a differential LoRA enhancement trained end-to-end and fused into the base model. The method iteratively gathers high-quality image pairs, filters them with aesthetic and semantic checks, and learns the image differences through LoRA, yielding progressive gains in aesthetics and human-preference metrics while preserving text-image alignment and safety. This approach reduces reliance on costly manual labeling and provides a scalable path to improved image quality without increasing inference cost, making it practically impactful for deploying higher-quality diffusion models.

Abstract

The emergence of diffusion models has significantly advanced image synthesis. The recent studies of model interaction and self-corrective reasoning approach in large language models offer new insights for enhancing text-to-image models. Inspired by these studies, we propose a novel method called ArtAug for enhancing text-to-image models in this paper. To the best of our knowledge, ArtAug is the first one that improves image synthesis models via model interactions with understanding models. In the interactions, we leverage human preferences implicitly learned by image understanding models to provide fine-grained suggestions for image synthesis models. The interactions can modify the image content to make it aesthetically pleasing, such as adjusting exposure, changing shooting angles, and adding atmospheric effects. The enhancements brought by the interaction are iteratively fused into the synthesis model itself through an additional enhancement module. This enables the synthesis model to directly produce aesthetically pleasing images without any extra computational cost. In the experiments, we train the ArtAug enhancement module on existing text-to-image models. Various evaluation metrics consistently demonstrate that ArtAug enhances the generative capabilities of text-to-image models without incurring additional computational costs. The source code and models will be released publicly.

Figures (8)

• Figure 1: Image examples improved by ArtAug. The base text-to-image model is FLUX.1[dev]. The ArtAug enhancement module is fused into the base model, without requiring additional computational resources.
• Figure 2: The framework of ArtAug encompasses three key components: the interaction algorithm, data generation and filtering, and differential training. This enhancement process can be iteratively applied to the model, facilitating iterative improvement.
• Figure 3: An example of the synthesis-understanding interaction process. By utilizing image understanding models to analyze and generate fine-grained prompts, we can enhance the overall quality of generated images.
• Figure 4: Statistical information of image pairs generated during the iterative training process. These statistical metrics are calculated based on the refined prompt utilized during the training process and are distinct from those presented in Table \ref{['tab:quantitative']}.
• Figure 5: Comparison of naive prompt refining and our interaction algorithm.