MoViE: Mobile Diffusion for Video Editing

TL;DR

This work addresses the prohibitive computational cost of diffusion-based video editing on edge devices by introducing MoViE, a mobile-optimized video editing pipeline. It combines a lightweight base model (Mobile-Pix2Pix) with multimodal guidance distillation to fuse text and image conditioning into a single forward pass, and an adversarial step distillation that collapses multiple diffusion steps without sacrificing controllability. On a $512\times384$ mobile pipeline, MoViE achieves about $12$–$12.5$ frames per second for long videos, with substantial reductions in FLOPs and latency while preserving editing quality. The approach enables practical, privacy-preserving, on-device video editing suitable for real-time applications on consumer hardware.

Abstract

Recent progress in diffusion-based video editing has shown remarkable potential for practical applications. However, these methods remain prohibitively expensive and challenging to deploy on mobile devices. In this study, we introduce a series of optimizations that render mobile video editing feasible. Building upon the existing image editing model, we first optimize its architecture and incorporate a lightweight autoencoder. Subsequently, we extend classifier-free guidance distillation to multiple modalities, resulting in a threefold on-device speedup. Finally, we reduce the number of sampling steps to one by introducing a novel adversarial distillation scheme which preserves the controllability of the editing process. Collectively, these optimizations enable video editing at 12 frames per second on mobile devices, while maintaining high quality. Our results are available at https://qualcomm-ai-research.github.io/mobile-video-editing/

Figures (11)

• Figure 1: MoViE is a fast video editing model, capable of generating $12$ frames per second on a mobile phone. It requires significantly fewer floating point operations (FLOPs) to edit a single video frame, making it computationally more efficient than competing methods.
• Figure 2: Multimodal Guidance Distillation Overview: Standard classifier-free guidance inference pipeline (left) with two input conditionings (image and text) requires three inference runs per diffusion step. Our distilled pipeline (right) incorporates guidance scales $s_I$ and $s_T$ into UNet and only performs one inference run.
• Figure 3: Adversarial Distillation: We distill a multi-step teacher into a single step student using adversarial losses. Unlike existing adversarial distillation approaches sauer2024fastzhang2024sf that forego guidance flexibility for faster sampling, we preserve guidance strength property during adversarial training by providing the synthetic latent $x_t$ from teacher's denoising process and conditioning the student on the corresponding guidance scales.
• Figure 4: MoViE at text guidance $[4.0,8.0,12.0]$ and image guidance $[1.25, 1.75]$. Our adversarial training maintains guidance scales, allowing us to control edit strength during inference. (Prompt: In Van Gogh Style)
• Figure 5: CLIP metrics for InstructPix2Pix, Mobile-Pix2pix, Multi-modal Guidance (MMG) Mobile-Pix2pix and MoViE. As shown in the graphs, proposed optimizations improve the efficiency greatly with minimum quality drop.