AMD-Hummingbird: Towards an Efficient Text-to-Video Model

TL;DR

This work tackles the efficiency gap in diffusion-based text-to-video models by introducing AMD-Hummingbird, a lightweight T2V framework that reduces the U-Net to 0.7B parameters and employs a two-stage distillation. The first stage prunes the network to remove middle blocks and halve per-layer blocks, while the second stage leverages reward-feedback learning from a mix of image-text and video-text rewards to sharpen visual fidelity and temporal coherence, all trained on publicly available WebVid-10M data. A novel data-processing pipeline uses Video Quality Assessment and LLM-based prompt recaption to curate high-quality video-text samples, improving training effectiveness. Empirically, Hummingbird achieves a 31-fold speedup over VideoCrafter2 on AMD Instinct MI250 and supports 26-frame videos, with top performance on VBench, demonstrating practical deployment potential on edge devices and AMD hardware.

Abstract

Text-to-Video (T2V) generation has attracted significant attention for its ability to synthesize realistic videos from textual descriptions. However, existing models struggle to balance computational efficiency and high visual quality, particularly on resource-limited devices, e.g.,iGPUs and mobile phones. Most prior work prioritizes visual fidelity while overlooking the need for smaller, more efficient models suitable for real-world deployment. To address this challenge, we propose a lightweight T2V framework, termed Hummingbird, which prunes existing models and enhances visual quality through visual feedback learning. Our approach reduces the size of the U-Net from 1.4 billion to 0.7 billion parameters, significantly improving efficiency while preserving high-quality video generation. Additionally, we introduce a novel data processing pipeline that leverages Large Language Models (LLMs) and Video Quality Assessment (VQA) models to enhance the quality of both text prompts and video data. To support user-driven training and style customization, we publicly release the full training code, including data processing and model training. Extensive experiments show that our method achieves a 31X speedup compared to state-of-the-art models such as VideoCrafter2, while also attaining the highest overall score on VBench. Moreover, our method supports the generation of videos with up to 26 frames, addressing the limitations of existing U-Net-based methods in long video generation. Notably, the entire training process requires only four GPUs, yet delivers performance competitive with existing leading methods. Hummingbird presents a practical and efficient solution for T2V generation, combining high performance, scalability, and flexibility for real-world applications.

Figures (3)

• Figure 1: Illustration of the proposed data processing pipeline, which includes video quality assessment, motion filtering, and prompt re-captioning using large language models to improve training data quality.
• Figure 2: Illustration of the proposed two-stage T2V diffusion model distillation pipeline. The first stage prunes the model's parameters to improve efficiency, while the second stage enhances visual quality through feedback learning.
• Figure 3: Qualitative Results. Video results generated from various text prompts. The proposed Hummingbird model is capable of generating high-quality visual results while accurately following the text prompts and producing semantically consistent content. We recommend viewing the video results using Adobe Acrobat Reader.