Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors

Yabo Zhang, Xinpeng Zhou, Yihan Zeng, Hang Xu, Hui Li, Wangmeng Zuo

TL;DR

FramePainter addresses the challenge of interactive image editing by leveraging video diffusion priors through an image-to-video formulation, enabling a source image and a lightweight editing cue to generate a two-frame video. A sparse control encoder injects edits into Stable Video Diffusion, while a novel matching attention branch enforces dense correspondence between edited and source tokens, mitigating limitations of temporal attention for large motions. The model is trained on thousands of video-derived image pairs and demonstrates superior editing plausibility, efficiency, and out-of-domain generalization, outperforming baselines with far less training data. This approach suggests a practical pathway for video-informed image editing tasks, reducing data and computation costs while delivering coherent, high-fidelity edits.

Abstract

Interactive image editing allows users to modify images through visual interaction operations such as drawing, clicking, and dragging. Existing methods construct such supervision signals from videos, as they capture how objects change with various physical interactions. However, these models are usually built upon text-to-image diffusion models, so necessitate (i) massive training samples and (ii) an additional reference encoder to learn real-world dynamics and visual consistency. In this paper, we reformulate this task as an image-to-video generation problem, so that inherit powerful video diffusion priors to reduce training costs and ensure temporal consistency. Specifically, we introduce FramePainter as an efficient instantiation of this formulation. Initialized with Stable Video Diffusion, it only uses a lightweight sparse control encoder to inject editing signals. Considering the limitations of temporal attention in handling large motion between two frames, we further propose matching attention to enlarge the receptive field while encouraging dense correspondence between edited and source image tokens. We highlight the effectiveness and efficiency of FramePainter across various of editing signals: it domainantly outperforms previous state-of-the-art methods with far less training data, achieving highly seamless and coherent editing of images, \eg, automatically adjust the reflection of the cup. Moreover, FramePainter also exhibits exceptional generalization in scenarios not present in real-world videos, \eg, transform the clownfish into shark-like shape. Our code will be available at https://github.com/YBYBZhang/FramePainter.

FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors

TL;DR

FramePainter addresses the challenge of interactive image editing by leveraging video diffusion priors through an image-to-video formulation, enabling a source image and a lightweight editing cue to generate a two-frame video. A sparse control encoder injects edits into Stable Video Diffusion, while a novel matching attention branch enforces dense correspondence between edited and source tokens, mitigating limitations of temporal attention for large motions. The model is trained on thousands of video-derived image pairs and demonstrates superior editing plausibility, efficiency, and out-of-domain generalization, outperforming baselines with far less training data. This approach suggests a practical pathway for video-informed image editing tasks, reducing data and computation costs while delivering coherent, high-fidelity edits.

Abstract

Interactive image editing allows users to modify images through visual interaction operations such as drawing, clicking, and dragging. Existing methods construct such supervision signals from videos, as they capture how objects change with various physical interactions. However, these models are usually built upon text-to-image diffusion models, so necessitate (i) massive training samples and (ii) an additional reference encoder to learn real-world dynamics and visual consistency. In this paper, we reformulate this task as an image-to-video generation problem, so that inherit powerful video diffusion priors to reduce training costs and ensure temporal consistency. Specifically, we introduce FramePainter as an efficient instantiation of this formulation. Initialized with Stable Video Diffusion, it only uses a lightweight sparse control encoder to inject editing signals. Considering the limitations of temporal attention in handling large motion between two frames, we further propose matching attention to enlarge the receptive field while encouraging dense correspondence between edited and source image tokens. We highlight the effectiveness and efficiency of FramePainter across various of editing signals: it domainantly outperforms previous state-of-the-art methods with far less training data, achieving highly seamless and coherent editing of images, \eg, automatically adjust the reflection of the cup. Moreover, FramePainter also exhibits exceptional generalization in scenarios not present in real-world videos, \eg, transform the clownfish into shark-like shape. Our code will be available at https://github.com/YBYBZhang/FramePainter.
Paper Structure (14 sections, 6 equations, 12 figures, 4 tables)

This paper contains 14 sections, 6 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Image Editing as an Image-to-Video Task
  5. Matching Attention for Dense Correspondence
  6. Constructing Samples from Video Data
  7. Experiments
  8. Experimental Setup
  9. Comparisons with Baselines
  10. Emerging Capabilities of FramePainter
  11. Ablation Study
  12. Conclusion
  13. Implementation Details of Different Visual Editing Instructions.
  14. More Visualizations and Comparisons.

Figures (12)

  • Figure 1: Examples of FramePainter. FramePainter allows users to manipulate images through intuitive visual instructions like drawing sketches, clicking points, and dragging regions. Benefiting from powerful video diffusion priors, it not only enables intuitive and plausible edits in common scenarios (e.g., adjust the reflection of the cup in red box), but also exhibits exceptional generalization in out-of-domain cases, e.g., transform the clownfish into shark-like shape.
  • Figure 2: Overview of FramePainter. Reformulating image editing as an image-to-video generation task, FramePainter takes a source image and an editing instruction as the first frame and control guidance, and produces a two-frame video comprising of reconstructed and target images. To improve visual consistency of two images involving large motion, matching attention is proposed to enlarge the receptive field and encourage dense correspondence between target and source image tokens.
  • Figure 3: Collected samples from videos. We present three types of editing signals from top to bottom: drawing sketches, click points, and dragging regions.
  • Figure 4: Qualitative comparisons across different visual editing instructions. Compared to the baselines, FramePainter not only achieves more coherent and plausible editing results, but also automatically polishes the edited images to meet real-world dynamics, e.g., remove duplicate tail and adjust car door in mirror (highlighted in red box). We note that LightningDrag and DragDiffusion require users to provide additional masks, whereas FramePainter does not.
  • Figure 5: Emerging capabilities of FramePainter. Although FramePainter is trained on image pairs from real-world videos, it demonstrates several emerging capabilities as a convenient tool: (i) Supporting highly intuitive and simplified instructions. (ii) Offering precise control over complex editing signals. (iii) Generalizing well to out-of-domain cases, such as shape transformation.
  • ...and 7 more figures