MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation

TL;DR

This paper proposes MoFu, a unified framework that significantly outperforms existing methods in preserving natural scale, subject fidelity, and overall visual quality and establishes a dedicated benchmark with controlled variations in subject scale and reference permutation.

Abstract

Multi-subject video generation aims to synthesize videos from textual prompts and multiple reference images, ensuring that each subject preserves natural scale and visual fidelity. However, current methods face two challenges: scale inconsistency, where variations in subject size lead to unnatural generation, and permutation sensitivity, where the order of reference inputs causes subject distortion. In this paper, we propose MoFu, a unified framework that tackles both challenges. For scale inconsistency, we introduce Scale-Aware Modulation (SMO), an LLM-guided module that extracts implicit scale cues from the prompt and modulates features to ensure consistent subject sizes. To address permutation sensitivity, we present a simple yet effective Fourier Fusion strategy that processes the frequency information of reference features via the Fast Fourier Transform to produce a unified representation. Besides, we design a Scale-Permutation Stability Loss to jointly encourage scale-consistent and permutation-invariant generation. To further evaluate these challenges, we establish a dedicated benchmark with controlled variations in subject scale and reference permutation. Extensive experiments demonstrate that MoFu significantly outperforms existing methods in preserving natural scale, subject fidelity, and overall visual quality.

Figures (11)

• Figure 1: Illustration of the challenges in multi-subject video generation. Existing models often suffer from scale inconsistency and permutation sensitivity. Our MoFu framework addresses both issues, producing natural-scale and permutation-invariant videos.
• Figure 2: Overview of the MoFu framework. MoFu integrates Scale-Aware Modulation (SMO), Fourier Fusion, and the Scale-Permutation Stability Loss (SPSL) into a DiT backbone. SMO extracts scale cues from the prompt via an LLM and adaptively modulates scale features to maintain natural subject scales. Fourier Fusion aggregates reference features in the frequency domain to form a permutation-invariant representation. Furthermore, SPSL jointly enforces scale consistency and permutation-invariant generation.
• Figure 3: Construction pipeline of the dataset and benchmark. We first perform video preprocessing and captioning, and multi-subject extraction and filtering. Multi-scale data for subjects and faces are then generated and refined, resulting in a dataset tailored for evaluating scale consistency and permutation-invariance in multi-subject video generation.
• Figure 4: Qualitative evaluation results of our method on different cases. Our model consistently generates video that maintains natural scale and permutation-invariance while accurately following the input text prompt. In row 5, the small rectangles overlaid on the reference images represent the relative area occupied by each subject in the frame. Specifically, the blue regions denote the padded white background, while the red regions indicate the actual subject area.
• Figure 5: Qualitative comparison on scale-inconsistent scenarios. In the left case, Phantom fails to maintain the correct relative scale of the bowl, while our approach produces a proportionally accurate result. Similarly, in the right case, the large monster truck is generated at a natural size relative to its environment.