On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices

TL;DR

On-device Sora tackles the problem of running diffusion-based text-to-video directly on mobile devices without retraining. It introduces three techniques—Linear Proportional Leap (LPL) to truncate denoising with Rectified Flow, Temporal Dimension Token Merging (TDTM) to cut temporal attention, and Concurrent Inference with Dynamic Loading (CI-DL) to fit large models into limited memory. Together, these methods enable on-device video generation with quality close to Open-Sora on high-end GPUs while delivering substantial speedups and eliminating the need for training or re-optimizing models. The approach, implemented on the iPhone 15 Pro and validated with VBench across multiple datasets, holds promise for democratizing access to state-of-the-art generative video on commodity devices and can be extended to other diffusion-based models.

Abstract

We present On-device Sora, the first model training-free solution for diffusion-based on-device text-to-video generation that operates efficiently on smartphone-grade devices. To address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices, the proposed On-device Sora applies three novel techniques to pre-trained video generative models. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) minimizes intensive token-processing computation in attention layers by merging consecutive tokens along the temporal dimension. Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions large models into smaller blocks and loads them into memory for concurrent model inference, effectively addressing the challenges of limited device memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental evaluations show that it is capable of generating high-quality videos on the device, comparable to those produced by high-end GPUs. These results show that On-device Sora enables efficient and high-quality video generation on resource-constrained mobile devices. We envision the proposed On-device Sora as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation on commodity mobile and embedded devices without resource-intensive re-training for model optimization (compression). The code implementation is available at a GitHub repository(https://github.com/eai-lab/On-device-Sora).

Figures (15)

• Figure 1: On-device Sora enables training-free text-to-video generation directly on the device by employing three key methods: 1) Linear Proportional Leap (LPL), 2) Temporal Dimension Token Merging (TDTM), and 3) Concurrent Inference with Dynamic Loading (CI-DL).
• Figure 2: The size of Open-Sora models: T5 raffel2020exploring (18.00 GB), STDiT opensora (4.50 GB), and VAE doersch2016tutorial (0.82 GB), which exceeds the available memory capacity of the iPhone 15 Pro apple2023 (3.3 GB).
• Figure 3: An abstracted illustration of trajectories and latent visualizations for $K=30$ and $n=15$: (a) Rectified Flow liu2022flow with full $k=30$ denoising steps, generating intact and complete video data, (b) Rectified Flow liu2022flow with $n+1=16$ denoising steps without applying Linear Proportional Leap, resulting in low-quality video data generation from variance with high step sizes ($dt_{k}$), and (c) Linear Proportional Leap with $n+1=15+1$ denoising steps, producing video data nearly equivalent to (a).
• Figure 4: An example of cosine similarities between two adjacent drifts estimated from STDiT opensora, i.e., $\boldsymbol{v}(P_{n},t_{n})$ and $\boldsymbol{v}(P_{n-1},t_{n-1})$ for 30 (red) and 50 steps (blue).
• Figure 5: In attention layers of STDiT opensora, two consecutive tokens are merged along the temporal dimension and subsequently unmerged after processing, reducing the token size by half and the computational complexity up to a quarter.