Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC

TL;DR

This paper tackles the challenge of adapting large pretrained diffusion models to diverse downstream image-generation tasks without task-specific networks. It introduces ONE-PIC, which uses In-Visual-Context Tuning to assemble a visual context from reference images and a Masking Strategy to recast downstream objectives as masked region prediction, all within the pretrained framework. The method leverages LoRA-based fine-tuning on a frozen SDXL backbone, learning only a small fraction of parameters, and shows faster convergence with lower computational costs across visual-conditional control, DreamBooth, image editing, and virtual try-on tasks. The results indicate that a general, context-driven fine-tuning approach can match or exceed task-specific methods while reducing design complexity and resource requirements, with practical implications for broader accessibility and deployment of diffusion-based systems.

Abstract

Large pretrained diffusion models have demonstrated impressive generation capabilities and have been adapted to various downstream tasks. However, unlike Large Language Models (LLMs) that can learn multiple tasks in a single model based on instructed data, diffusion models always require additional branches, task-specific training strategies, and losses for effective adaptation to different downstream tasks. This task-specific fine-tuning approach brings two drawbacks. 1) The task-specific additional networks create gaps between pretraining and fine-tuning which hinders the transfer of pretrained knowledge. 2) It necessitates careful additional network design, raising the barrier to learning and implementation, and making it less user-friendly. Thus, a question arises: Can we achieve a simple, efficient, and general approach to fine-tune diffusion models? To this end, we propose ONE-PIC. It enhances the inherited generative ability in the pretrained diffusion models without introducing additional modules. Specifically, we propose In-Visual-Context Tuning, which constructs task-specific training data by arranging source images and target images into a single image. This approach makes downstream fine-tuning closer to the pertaining, allowing our model to adapt more quickly to various downstream tasks. Moreover, we propose a Masking Strategy to unify different generative tasks. This strategy transforms various downstream fine-tuning tasks into predictions of the masked portions. The extensive experimental results demonstrate that our method is simple and efficient which streamlines the adaptation process and achieves excellent performance with lower costs. Code is available at https://github.com/tobran/ONE-PIC.

Figures (11)

• Figure 1: Our ONE-PIC provides a simple and versatile framework for fine-tuning across various downstream image generation tasks.
• Figure 2: (a) The current adaptation process for downstream fine-tuning models in diffusion models. (b) Our ONE-PIC significantly simplifies the network design and fine-tuning process.
• Figure 3: (a) Pretrained large language models can be applied to text classification tasks through appropriate text contexts. (b) The idea is that diffusion models can utilize visual contexts to adapt to downstream image generation tasks.
• Figure 4: The architecture of ONE-PIC for different downstream tasks. We show the visual context of visual conditional control, virtual try-on, and fashion extraction.
• Figure 5: In visual conditional control generation tasks, different visual contexts can have varying impacts on the results.