Real-Time Person Image Synthesis Using a Flow Matching Model

TL;DR

The paper presents Real-Time Flow Matching (RPFM), a latent-space, transformer-based flow matching approach for Pose-Guided Person Image Synthesis that conditions on a source image and a target pose to enable near real-time generation. By mapping images to a latent representation via a VAE and predicting the latent flow $dz/dt$ with a DiT backbone, the method reduces sampling steps while maintaining competitive visual fidelity. Two conditioning mechanisms—input concatenation and conditional aggregation—are shown to improve speed and quality, with classifier-free guidance further enhancing sampling behavior. Experiments on the DeepFashion dataset demonstrate near real-time sampling with performance on par with state-of-the-art diffusion-based models, highlighting the method's potential for real-time video applications such as sign language synthesis. Limitations include incomplete ablation at the largest scale due to resource constraints and a need for deeper analysis of conditioning strategies across broader NFEs.

Abstract

Pose-Guided Person Image Synthesis (PGPIS) generates realistic person images conditioned on a target pose and a source image. This task plays a key role in various real-world applications, such as sign language video generation, AR/VR, gaming, and live streaming. In these scenarios, real-time PGPIS is critical for providing immediate visual feedback and maintaining user immersion.However, achieving real-time performance remains a significant challenge due to the complexity of synthesizing high-fidelity images from diverse and dynamic human poses. Recent diffusion-based methods have shown impressive image quality in PGPIS, but their slow sampling speeds hinder deployment in time-sensitive applications. This latency is particularly problematic in tasks like generating sign language videos during live broadcasts, where rapid image updates are required. Therefore, developing a fast and reliable PGPIS model is a crucial step toward enabling real-time interactive systems. To address this challenge, we propose a generative model based on flow matching (FM). Our approach enables faster, more stable, and more efficient training and sampling. Furthermore, the proposed model supports conditional generation and can operate in latent space, making it especially suitable for real-time PGPIS applications where both speed and quality are critical. We evaluate our proposed method, Real-Time Person Image Synthesis Using a Flow Matching Model (RPFM), on the widely used DeepFashion dataset for PGPIS tasks. Our results show that RPFM achieves near-real-time sampling speeds while maintaining performance comparable to the state-of-the-art models. Our methodology trades off a slight, acceptable decrease in generated-image accuracy for over a twofold increase in generation speed, thereby ensuring real-time performance.

Figures (7)

• Figure 1: An illustration of the DeepFashion PGPIS dataset is presented. In this task, the individual depicted in the source image must be modified to align with the target pose.
• Figure 2: The architecture of our Real-Time Person Image Synthesis employs a Flow Matching Model. This architecture encompasses both the input concatenation component and the conditional aggregation component, with the output represented as $dz/dt$. It is important to note that certain VAE components have been excluded from the architectural diagram, specifically the inputs and conditions intended for concatenation at the input concatenation stage. The orange arrow indicates input concatenation.
• Figure 3: visualization of \ref{['tab:speed']}
• Figure 4: Our model produces images that correspond to the intended pose while maintaining the characteristics of the source image.
• Figure 5: A qualitative comparative study with baselines (in the DeepFashion dataset)