DragLoRA: Online Optimization of LoRA Adapters for Drag-based Image Editing in Diffusion Model

TL;DR

DragLoRA addresses the precision-efficiency trade-off in drag-based diffusion editing by online-optimizing LoRA adapters instead of latent features. It introduces a dual-objective loss combining drag-guided deformation with a delta denoising score (DDS) regularization, plus a cyclic input latent feature adaptation (ILFA) to stabilize progress. An adaptive optimization scheme (ASS) toggles between motion supervision and input adaptation, while Efficient Point Tracking (EPT) narrows candidate regions for handle-point updates. Across qualitative and quantitative benchmarks, DragLoRA achieves state-of-the-art edit fidelity with lower computation time, demonstrating practical utility for interactive, point-driven image editing in diffusion models. The approach can extend to region-based edits and reference-image guidance, with potential for broader applicability to online model adaptation in generative editing workflows.

Abstract

Drag-based editing within pretrained diffusion model provides a precise and flexible way to manipulate foreground objects. Traditional methods optimize the input feature obtained from DDIM inversion directly, adjusting them iteratively to guide handle points towards target locations. However, these approaches often suffer from limited accuracy due to the low representation ability of the feature in motion supervision, as well as inefficiencies caused by the large search space required for point tracking. To address these limitations, we present DragLoRA, a novel framework that integrates LoRA (Low-Rank Adaptation) adapters into the drag-based editing pipeline. To enhance the training of LoRA adapters, we introduce an additional denoising score distillation loss which regularizes the online model by aligning its output with that of the original model. Additionally, we improve the consistency of motion supervision by adapting the input features using the updated LoRA, giving a more stable and accurate input feature for subsequent operations. Building on this, we design an adaptive optimization scheme that dynamically toggles between two modes, prioritizing efficiency without compromising precision. Extensive experiments demonstrate that DragLoRA significantly enhances the control precision and computational efficiency for drag-based image editing. The Codes of DragLoRA are available at: https://github.com/Sylvie-X/DragLoRA.

Figures (10)

• Figure 1: Visual comparison between DragDiffusion shi2024dragdiffusion and DragLoRA at each step. For a given image and user request, we present dragged images at four intermediate steps. In DragLoRA, the steps are divided into two types: DOO+ILFA (Red) or ILFA-only (Blue), where DOO stands for Dual-Objective Optimization and ILFA stands for Input Latent Feature Adaptation. DragDiffusion requires 80 optimization steps but produces results with lower fidelity. In contrast, DragLoRA achieves more precise deformations with fewer optimization steps and more total steps, which consumes less time thanks to the high-efficiency of ILFA. The middle $minD$ curves demonstrate that DragLoRA achieves better point tracking, preserving source details more accurately than DragDiffusion.
• Figure 2: Overview of our proposed DragLoRA. Given an inversion code $z_t$ at $t = 35$ from a source image $x_0$, we incorporate a LoRA adapter and optimize it online using $L_\text{drag}$ and $L_\text{DDS}$. $L_\text{drag}$ primarily guides handle point movement, while $L_\text{DDS}$ constrains LoRA to remain close to the original model, preserving the fidelity of the edited image. Additionally, the input feature to the UNet undergoes a denoise-renoise cycle in the foreground region, allowing the LoRA adapter to drive handle points even without gradient updates. DragLoRA dynamically switches between motion supervision and input-adaptation-only modes based on point tracking quality, ensuring both efficiency and stability in the editing process.
• Figure 3: Comparison among different point tracking schemes. (a) A common strategy locates new point in a square neighborhood around current handle point. (b) and (c) reduces the search region using target point, only grid points in the intersection region are considered as candidate. (d) uses a linear line and needs feature interpolation. (b) and (d) are initially proposed in jiang2024clipdrag and ling2024freedrag, respectively.
• Figure 4: Qualitative comparisons with shi2024dragdiffusionliu2024dragnoisezhang2024gooddrag. The proposed DragLoRA outperforms existing approaches in both perceptual quality and the accuracy of drag editing.
• Figure 5: Comparisons in Drag-Back pipeline. Following ling2024freedrag, we perform two symmetric drag edits: the first adheres to the input drag annotation and the second reverses it. We focus on the similarity between the drag-back image and the input to validate image fidelity and edit accuracy.