InstructDiffusion: A Generalist Modeling Interface for Vision Tasks

TL;DR

This work introduces InstructDiffusion, a diffusion-based generalist interface that casts diverse vision tasks into instruction-guided image manipulation in pixel space, covering keypoint detection, segmentation, editing, and enhancement. It employs a three-stage training pipeline—pretraining adaptation, task-specific training, and instruction tuning—along with diverse datasets and a new Image Editing in the Wild (IEIW) resource to enable broad task coverage and open-set generalization. By using detailed natural-language-like instructions rather than simple task indicators, the model achieves strong performance across multiple tasks and demonstrates AGI-like capabilities on unseen datasets. The results substantiate the potential of a unified, instruction-driven vision framework and point to future work in richer representations and self-supervised learning to further improve generalization.

Abstract

We present InstructDiffusion, a unifying and generic framework for aligning computer vision tasks with human instructions. Unlike existing approaches that integrate prior knowledge and pre-define the output space (e.g., categories and coordinates) for each vision task, we cast diverse vision tasks into a human-intuitive image-manipulating process whose output space is a flexible and interactive pixel space. Concretely, the model is built upon the diffusion process and is trained to predict pixels according to user instructions, such as encircling the man's left shoulder in red or applying a blue mask to the left car. InstructDiffusion could handle a variety of vision tasks, including understanding tasks (such as segmentation and keypoint detection) and generative tasks (such as editing and enhancement). It even exhibits the ability to handle unseen tasks and outperforms prior methods on novel datasets. This represents a significant step towards a generalist modeling interface for vision tasks, advancing artificial general intelligence in the field of computer vision.

Figures (11)

• Figure 1: We introduce InstructDiffusion, a generalist modeling interface for vision tasks. Given input image and human instruction, our unified model effectively accomplishes tasks such as image editing, segmentation, keypoint estimation, detection, and low-level vision.
• Figure 2: Training pipeline of our method. To illustrate concisely, we take keypoint detection as an example.
• Figure 3: The keypoint detection results generated by our model. The instructions are as follows: (a) Mark the car logo with a blue circle. (b) Put a blue circle on the nose of the white tiger and use the red color to draw a circle around the left shoulder of the white tiger. (c) Create a yellow circle around the right eye of the whale. (d) Use the color blue to encircle the right wrist of the person on the far left and draw a yellow circle over the left wrist of the person on the far right.
• Figure 4: The segmentation results generated by our model. The instructions are as follows: (a) Mark the pixels of cat in the mirror to blue and leave the rest unchanged. (b) Fill in the pixels of neutrophil with yellow, retaining the existing colors of the remaining pixels. (c) Modify the pixels of Oriental Pearl Tower to red without affecting any other pixels. (d) Paint the pixels of shadow in blue and maintain the current appearance of the other pixels.
• Figure 5: InstructDiffusion is also applicable to low-level vision tasks, including image deblurring, denoising, and watermark removal.