GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter

TL;DR

The paper tackles the scarcity of instruction-based image-editing data by introducing GalaxyEdit, a large-scale dataset generated via an automated pipeline from COCO that supports diverse add/remove edits with multiple instructions. Fine-tuning Stable Diffusion v1.5 on GalaxyEdit yields strong improvements over baselines in add/remove tasks, and Generalization to external datasets like MagicBrush is demonstrated. To enable on-device usage, the authors propose ControlNet-Vxs, a Volterra-filter-based non-linear fusion between a frozen base model and a control network, surpassing ControlNet-xs in both add/remove and canny-guided generation. Together, GalaxyEdit and the Volterra-based adapter offer improved instruction-following fidelity and generalization, with future work pointing toward richer 3D spatial reasoning and broader applicability of non-linear adapters.

Abstract

Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, suboptimal end-to-end models, data diversity constraints and high expenses. We propose an automated data generation pipeline aimed at alleviating such limitations, and introduce GalaxyEdit - a large-scale image editing dataset for add and remove operations. We fine-tune the SD v1.5 model on our dataset and find that our model can successfully handle a broader range of objects and complex editing instructions, outperforming state-of-the-art methods in FID scores by 11.2\% and 26.1\% for add and remove tasks respectively. Furthermore, in light of on-device usage scenarios, we expand our research to include task-specific lightweight adapters leveraging the ControlNet-xs architecture. While ControlNet-xs excels in canny and depth guided generation, we propose to improve the communication between the control network and U-Net for more intricate add and remove tasks. We achieve this by enhancing ControlNet-xs with non-linear interaction layers based on Volterra filters. Our approach outperforms ControlNet-xs in both add/remove and canny-guided image generation tasks, highlighting the effectiveness of the proposed enhancement.

Figures (6)

• Figure 1: Visual illustration of outputs generated by our proposed model.
• Figure 2: Proposed data generation pipeline for GalaxyEdit dataset.
• Figure 3: ControlNet-Vxs Model Architecture.
• Figure 4: Qualitative results on Object Removal (Left) and Object Addition (Right). We compare our model against leading methods for add and remove showcasing diverse instruction following capability while maintaining high fidelity to the input image.
• Figure 5: Category-wise results on GalaxyEdit test set