DreamOmni3: Scribble-based Editing and Generation

TL;DR

DreamOmni3 addresses the limits of language-only image editing by introducing scribble-based editing and generation to a unified generation-editing framework. It builds a data synthesis pipeline on top of DreamOmni2 with Referseg-guided scribble placement, and a joint-input scheme that feeds both original and scribbled images with shared encodings to handle multi-region edits. A dedicated DreamOmni3 benchmark based on real-world images evaluates editing and generation using VLM- and human-based criteria, showing state-of-the-art performance and robust generalization. The work enables GUI-style, scribble-assisted creative editing and releases models and code for broad adoption.

Abstract

Recently unified generation and editing models have achieved remarkable success with their impressive performance. These models rely mainly on text prompts for instruction-based editing and generation, but language often fails to capture users intended edit locations and fine-grained visual details. To this end, we propose two tasks: scribble-based editing and generation, that enables more flexible creation on graphical user interface (GUI) combining user textual, images, and freehand sketches. We introduce DreamOmni3, tackling two challenges: data creation and framework design. Our data synthesis pipeline includes two parts: scribble-based editing and generation. For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and multimodal instruction-based editing, image fusion, and doodle editing. Based on DreamOmni2 dataset, we extract editable regions and overlay hand-drawn boxes, circles, doodles or cropped image to construct training data. For scribble-based generation, we define three tasks: scribble and instruction-based generation, scribble and multimodal instruction-based generation, and doodle generation, following similar data creation pipelines. For the framework, instead of using binary masks, which struggle with complex edits involving multiple scribbles, images, and instructions, we propose a joint input scheme that feeds both the original and scribbled source images into the model, using different colors to distinguish regions and simplify processing. By applying the same index and position encodings to both images, the model can precisely localize scribbled regions while maintaining accurate editing. Finally, we establish comprehensive benchmarks for these tasks to promote further research. Experimental results demonstrate that DreamOmni3 achieves outstanding performance, and models and code will be publicly released.

Figures (4)

• Figure 1: The gallery of DreamOmni3, which has scribble-based editing and generation capabilities.
• Figure 2: The overview of DreamOmni3's training data construction and framework. The overview of DreamOmni3's training data construction and framework: (a) We create scribble-based editing training data. For scribble and multimodal instruction-based editing, we use Referseg to locate edit objects and paste corresponding scribbles onto the source and reference images to create training pairs. For scribble and instruction-based editing, we omit the reference image. For doodle editing, we use a dedicated model to convert the edit objects into abstract sketches and paste them back into the source image. For image fusion, we crop objects from the reference image and paste them into the corresponding position on the source image to build training pairs. (b) Scribble-based generation training data is created similarly to editing, except the source image is replaced with a blank white canvas. (c) DreamOmni3 builds on the framework of DreamOmni2 dreamomni2, introducing a joint input scheme for scribble inputs. We also apply the same encoding scheme to both the source and scribbled images, ensuring better pixel alignment and perfect compatibility with previous image and language instruction editing.
• Figure 3: Visual comparison of scribble-based editing. Compared to other competitive methods and even closed-source commercial models (GPT-4o and Nano Banana), DreamOmni3 shows more accurate editing results and better consistency.
• Figure 4: Visual comparison of scribble-based generation. Our DreamOmni3 significantly outperforms current open-source models and achieves generation results comparable to closed-source commercial models (GPT-4o and Nano Banana).