ProFashion: Prototype-guided Fashion Video Generation with Multiple Reference Images

TL;DR

ProFashion tackles the challenge of view-consistent fashion video generation conditioned on multiple reference images. It introduces a Reference Encoder to extract multi-scale reference features, a Pose-aware Prototype Aggregator (PPA) to form frame-wise prototypes, and a Flow-enhanced Prototype Instantiator (FPI) that leverages human keypoint motion flow within a spatiotemporal attention framework. Trained in two stages with denoising and offset losses, ProFashion demonstrates superior performance on MRFashion-7K and competitive results on UBC Fashion, significantly improving view consistency and motion coherence while maintaining computational efficiency close to single-reference methods. The approach holds practical promise for online fashion visualization by enabling richer portrayal of garments from multiple viewpoints. Overall, ProFashion advances multi-image conditioned diffusion-based video generation in the fashion domain with concrete architectural components that balance quality, consistency, and efficiency.

Abstract

Fashion video generation aims to synthesize temporally consistent videos from reference images of a designated character. Despite significant progress, existing diffusion-based methods only support a single reference image as input, severely limiting their capability to generate view-consistent fashion videos, especially when there are different patterns on the clothes from different perspectives. Moreover, the widely adopted motion module does not sufficiently model human body movement, leading to sub-optimal spatiotemporal consistency. To address these issues, we propose ProFashion, a fashion video generation framework leveraging multiple reference images to achieve improved view consistency and temporal coherency. To effectively leverage features from multiple reference images while maintaining a reasonable computational cost, we devise a Pose-aware Prototype Aggregator, which selects and aggregates global and fine-grained reference features according to pose information to form frame-wise prototypes, which serve as guidance in the denoising process. To further enhance motion consistency, we introduce a Flow-enhanced Prototype Instantiator, which exploits the human keypoint motion flow to guide an extra spatiotemporal attention process in the denoiser. To demonstrate the effectiveness of ProFashion, we extensively evaluate our method on the MRFashion-7K dataset we collected from the Internet. ProFashion also outperforms previous methods on the UBC Fashion dataset.

Figures (10)

• Figure 1: Single reference image fails to provide sufficient information when generating fashion videos for garments with view-dependent patterns and leads to severe hallucination. In contrast, multi-image-conditioned fashion video generation ensures satisfactory view consistency (§\ref{['sec:intro']}).
• Figure 2: Overall framework of ProFashion (§\ref{['sec:framework']}). It first converts the inputs into latent spaces with different encoders. Then, a Reference Encoder (§\ref{['sec:ref_enc']}) is used for extracting multi-scale representation of reference images. Next, PPA (§\ref{['sec:ppa']}) is adopted to aggregate the multi-scale representation and global features into fine-grained and global prototypes according to pose similarity. Finally, it utilizes FPI (§\ref{['sec:fpi']}) to conduct a prototype-guided iterative denoising process enhanced by keypoint motion flow.
• Figure 3: Details of PPA (§\ref{['sec:ppa']}). It first uses a pose-aware selector to calculate the prototype aggregation map and then conducts fine-grained and global prototype aggregation accordingly.
• Figure 4: Details of spatiotemporal attention in FTA (§\ref{['sec:fpi']}). It conducts multi-head attention with original frame-wise features as queries and resampled features with 1 frame's offset as keys and values. The resampling process is guided by the query-conditioned offset prediction supervised by human keypoint motion flow.
• Figure 5: Visualizations on the test split of MRFashion-7K (§\ref{['sec:mrfashion']}).