RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame Animated Sticker Generation

TL;DR

This work tackles resource-limited multi-frame animated sticker generation by shifting from large pre-trained models with parameter-efficient tuning to training a compact model from scratch on million-scale data. It introduces RDTF, combining Discrete Frame Generation Network, dual-mask data utilization, and difficulty-adaptive curriculum learning to exploit scarce data effectively. A key contribution is VSD2M, a 2.09M-sample vision-language dataset tailored for animated stickers, enabling robust training and evaluation. Empirical results show RDTF outperforms PEFT baselines on FVD, VQA, and CLIP similarity across ASG tasks and generalizes to other low-frame-rate video domains, highlighting a practical pathway for resource-constrained video generation.

Abstract

Recently, significant advancements have been achieved in video generation technology, but applying it to resource-constrained downstream tasks like multi-frame animated sticker generation (ASG) characterized by low frame rates, abstract semantics, and long tail frame length distribution-remains challenging. Parameter-efficient fine-tuning (PEFT) techniques (e.g., Adapter, LoRA) for large pre-trained models suffer from insufficient fitting ability and source-domain knowledge interference. In this paper, we propose Resource-Efficient Dual-Mask Training Framework (RDTF), a dedicated solution for multi-frame ASG task under resource constraints. We argue that training a compact model from scratch with million-level samples outperforms PEFT on large models, with RDTF realizing this via three core designs: 1) a Discrete Frame Generation Network (DFGN) optimized for low-frame-rate ASG, ensuring parameter efficiency; 2) a dual-mask based data utilization strategy to enhance the availability and diversity of limited data; 3) a difficulty-adaptive curriculum learning method that decomposes sample entropy into static and adaptive components, enabling easy-to-difficult training convergence. To provide high-quality data support for RDTFs training from scratch, we construct VSD2M-a million-level multi-modal animated sticker dataset with rich annotations (static and animated stickers, action-focused text descriptions)-filling the gap of dedicated animated data for ASG task. Experiments demonstrate that RDTF is quantitatively and qualitatively superior to state-of-the-art PEFT methods (e.g., I2V-Adapter, SimDA) on ASG tasks, verifying the feasibility of our framework under resource constraints.

Figures (7)

• Figure 1: Results of Animated Sticker Generation via RDTF. Our approach demonstrates excellent performance across diverse ASG tasks. Gray boxes signify either textual or visual guidance. See https://mails9523.github.io for dynamic results.
• Figure 2: Overview of resource-efficient dual-mask training framework. We propose a discrete frame generation network to model the discreteness between animated sticker frames. Furthermore, the dual masks, $i.e.$, condition mask and loss mask, are designed to improve the availability and expand the diversity of limited data. The difficulty-adaptive curriculum learning is applied to facilitate convergence.
• Figure 3: Frame extraction algorithm based on feature clustering. During training, data are clustered into $k$ clusters randomly to increase the information density.
• Figure 4: Frame distribution in collected sticker dataset, which follows the long-tail distribution, $i. e.$, more short frames and fewer long frames. It poses challenges for fully utilizing long-frame data, highlighting the necessity of our clustering-based frame extraction and dual-mask strategy to enhance data utilization efficiency.
• Figure 5: The masked frame length and task type during training are independent of each other, which makes it difficult to determine a route to obtain entropy-increasing samples stably.