MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers

TL;DR

The paper tackles open-domain instruction-guided image manipulation by diffusion models, introducing MoEController which unifies global and local editing through a three-expert mixture-of-experts controller. It constructs a large-scale global manipulation dataset via ChatGPT and ControlNet, and trains a conditional diffusion model with a cross-attention fusion module and a reconstruction loss to preserve image identity. The key contributions are the MOE architecture with task-specific experts, a dataset construction pipeline for global edits, and empirical demonstrations of state-of-the-art performance on both global and local manipulation tasks with strong zero-shot generalization. This work advances practical, instruction-based image editing by enabling flexible, open-domain transformations under diverse user instructions.

Abstract

Diffusion-model-based text-guided image generation has recently made astounding progress, producing fascinating results in open-domain image manipulation tasks. Few models, however, currently have complete zero-shot capabilities for both global and local image editing due to the complexity and diversity of image manipulation tasks. In this work, we propose a method with a mixture-of-expert (MOE) controllers to align the text-guided capacity of diffusion models with different kinds of human instructions, enabling our model to handle various open-domain image manipulation tasks with natural language instructions. First, we use large language models (ChatGPT) and conditional image synthesis models (ControlNet) to generate a large number of global image transfer dataset in addition to the instruction-based local image editing dataset. Then, using an MOE technique and task-specific adaptation training on a large-scale dataset, our conditional diffusion model can edit images globally and locally. Extensive experiments demonstrate that our approach performs surprisingly well on various image manipulation tasks when dealing with open-domain images and arbitrary human instructions. Please refer to our project page: [https://oppo-mente-lab.github.io/moe_controller/]

Figures (6)

• Figure 1: Arbitrary instruction-guided image global and local manipulation visualizations.
• Figure 2: Cross-attention heat map of the core words and generated image of different methods.
• Figure 3: An overview of our approach. Left: pipeline of dataset construction. Right: MOE controller structure.
• Figure 4: Comparison of image global and local manipulation tasks with and without mixture-of-experts.
• Figure 5: The effect of image style transfer with different w.