Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIBE: Visual Instruction Based Editor

Grigorii Alekseenko, Aleksandr Gordeev, Irina Tolstykh, Bulat Suleimanov, Vladimir Dokholyan, Georgii Fedorov, Sergey Yakubson, Aleksandra Tsybina, Mikhail Chernyshov, Maksim Kuprashevich

TL;DR

VIBE demonstrates that high-quality instruction-based image editing is achievable with a compact architecture by pairing a 2B visual language model with a 1.6B diffusion backbone. It grounds edits in the source image via a learnable meta-token and a lightweight connector, using channel-wise reference guidance to preserve throughput. A four-stage training pipeline—connector alignment, pretraining, supervised fine-tuning, and Diffusion-DPO—along with a multi-source, real-world data strategy and rigorous filtering, yields robust performance while maintaining strict source consistency. On ImgEdit and GEdit benchmarks, VIBE rivals heavier models, particularly in edits that require preservation of input content, while offering substantial gains in efficiency and real-world applicability. Limitations include difficulty with highly complex geometric edits and potential biases from pretrained components, pointing to directions like distillation, quantization, and expanded real-world data in future work.

Abstract

Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside highly capable commercial systems. However, only a limited number of open-source approaches currently achieve real-world quality. In addition, diffusion backbones, the dominant choice for these pipelines, are often large and computationally expensive for many deployments and research settings, with widely used variants typically containing 6B to 20B parameters. This paper presents a compact, high-throughput instruction-based image editing pipeline that uses a modern 2B-parameter Qwen3-VL model to guide the editing process and the 1.6B-parameter diffusion model Sana1.5 for image generation. Our design decisions across architecture, data processing, training configuration, and evaluation target low-cost inference and strict source consistency while maintaining high quality across the major edit categories feasible at this scale. Evaluated on the ImgEdit and GEdit benchmarks, the proposed method matches or exceeds the performance of substantially heavier baselines, including models with several times as many parameters and higher inference cost, and is particularly strong on edits that require preserving the input image, such as an attribute adjustment, object removal, background edits, and targeted replacement. The model fits within 24 GB of GPU memory and generates edited images at up to 2K resolution in approximately 4 seconds on an NVIDIA H100 in BF16, without additional inference optimizations or distillation.

VIBE: Visual Instruction Based Editor

TL;DR

VIBE demonstrates that high-quality instruction-based image editing is achievable with a compact architecture by pairing a 2B visual language model with a 1.6B diffusion backbone. It grounds edits in the source image via a learnable meta-token and a lightweight connector, using channel-wise reference guidance to preserve throughput. A four-stage training pipeline—connector alignment, pretraining, supervised fine-tuning, and Diffusion-DPO—along with a multi-source, real-world data strategy and rigorous filtering, yields robust performance while maintaining strict source consistency. On ImgEdit and GEdit benchmarks, VIBE rivals heavier models, particularly in edits that require preservation of input content, while offering substantial gains in efficiency and real-world applicability. Limitations include difficulty with highly complex geometric edits and potential biases from pretrained components, pointing to directions like distillation, quantization, and expanded real-world data in future work.

Abstract

Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside highly capable commercial systems. However, only a limited number of open-source approaches currently achieve real-world quality. In addition, diffusion backbones, the dominant choice for these pipelines, are often large and computationally expensive for many deployments and research settings, with widely used variants typically containing 6B to 20B parameters. This paper presents a compact, high-throughput instruction-based image editing pipeline that uses a modern 2B-parameter Qwen3-VL model to guide the editing process and the 1.6B-parameter diffusion model Sana1.5 for image generation. Our design decisions across architecture, data processing, training configuration, and evaluation target low-cost inference and strict source consistency while maintaining high quality across the major edit categories feasible at this scale. Evaluated on the ImgEdit and GEdit benchmarks, the proposed method matches or exceeds the performance of substantially heavier baselines, including models with several times as many parameters and higher inference cost, and is particularly strong on edits that require preserving the input image, such as an attribute adjustment, object removal, background edits, and targeted replacement. The model fits within 24 GB of GPU memory and generates edited images at up to 2K resolution in approximately 4 seconds on an NVIDIA H100 in BF16, without additional inference optimizations or distillation.
Paper Structure (64 sections, 7 equations, 11 figures, 5 tables)

This paper contains 64 sections, 7 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Production-oriented open editors and efficiency constraints.
  4. Architectures for Conditioning the Source Image
  5. Architectures for Interpreting Instructions
  6. Training Pipelines, Data, and Alignment
  7. Consistency and Real-World Instruction Distributions
  8. Method
  9. Reference Image Guidance
  10. Textual Guidance Based on VLM
  11. Interface between VLM and Diffusion model
  12. Connector Design
  13. Training Approach
  14. Connector Alignment
  15. T2I Data Injections
  16. ...and 49 more sections

Figures (11)

  • Figure 1: Illustrative examples of image edits generated by VIBE.
  • Figure 2: Illustrative examples of image edits generated by VIBE.
  • Figure 3: Model architecture.
  • Figure 4: Examples of UltraEdit. Top row is the original set, bottom row is the remastered version.
  • Figure 5: Example of background removal on the LVIS dataset. High-quality dataset annotations and carefully crafted engineering heuristics enable automatic instruction generation, making localization and object pointing somewhat tricky.
  • ...and 6 more figures