Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowAct-R1: Towards Interactive Humanoid Video Generation

Lizhen Wang, Yongming Zhu, Zhipeng Ge, Youwei Zheng, Longhao Zhang, Tianshu Hu, Shiyang Qin, Mingshuang Luo, Jiaxu Zhang, Xin Chen, Yulong Wang, Zerong Zheng, Jianwen Jiang, Chao Liang, Weifeng Chen, Xing Wang, Yuan Zhang, Mingyuan Gao

TL;DR

FlowAct-R1 tackles real-time interactive humanoid video generation by integrating a Seedance-based Multimodal Diffusion Transformer (MMDiT) with chunkwise diffusion forcing and a memory-based streaming framework. It enables streaming, arbitrary-length video at 25 FPS and 480p with a TTFF around 1.5 seconds, while maintaining high behavioral vividness through memory refinement and MLLM-guided action planning. The method combines a three-stage training curriculum, distillation to 3 NFEs, FP8 quantization, and system-level optimizations to deliver low-latency, high-fidelity, full-body control across diverse character styles. Experimental results demonstrate superior realism, natural transitions between interactive states, and robust generalization compared to state-of-the-art baselines, signaling practical potential for live streaming, virtual companionship, and real-time conferencing. Ethical considerations address potential misuse with an emphasis on access control and privacy-aware demonstrations, noting that generated human images were AI-generated to ensure compliance.

Abstract

Interactive humanoid video generation aims to synthesize lifelike visual agents that can engage with humans through continuous and responsive video. Despite recent advances in video synthesis, existing methods often grapple with the trade-off between high-fidelity synthesis and real-time interaction requirements. In this paper, we propose FlowAct-R1, a framework specifically designed for real-time interactive humanoid video generation. Built upon a MMDiT architecture, FlowAct-R1 enables the streaming synthesis of video with arbitrary durations while maintaining low-latency responsiveness. We introduce a chunkwise diffusion forcing strategy, complemented by a novel self-forcing variant, to alleviate error accumulation and ensure long-term temporal consistency during continuous interaction. By leveraging efficient distillation and system-level optimizations, our framework achieves a stable 25fps at 480p resolution with a time-to-first-frame (TTFF) of only around 1.5 seconds. The proposed method provides holistic and fine-grained full-body control, enabling the agent to transition naturally between diverse behavioral states in interactive scenarios. Experimental results demonstrate that FlowAct-R1 achieves exceptional behavioral vividness and perceptual realism, while maintaining robust generalization across diverse character styles.

FlowAct-R1: Towards Interactive Humanoid Video Generation

TL;DR

FlowAct-R1 tackles real-time interactive humanoid video generation by integrating a Seedance-based Multimodal Diffusion Transformer (MMDiT) with chunkwise diffusion forcing and a memory-based streaming framework. It enables streaming, arbitrary-length video at 25 FPS and 480p with a TTFF around 1.5 seconds, while maintaining high behavioral vividness through memory refinement and MLLM-guided action planning. The method combines a three-stage training curriculum, distillation to 3 NFEs, FP8 quantization, and system-level optimizations to deliver low-latency, high-fidelity, full-body control across diverse character styles. Experimental results demonstrate superior realism, natural transitions between interactive states, and robust generalization compared to state-of-the-art baselines, signaling practical potential for live streaming, virtual companionship, and real-time conferencing. Ethical considerations address potential misuse with an emphasis on access control and privacy-aware demonstrations, noting that generated human images were AI-generated to ensure compliance.

Abstract

Interactive humanoid video generation aims to synthesize lifelike visual agents that can engage with humans through continuous and responsive video. Despite recent advances in video synthesis, existing methods often grapple with the trade-off between high-fidelity synthesis and real-time interaction requirements. In this paper, we propose FlowAct-R1, a framework specifically designed for real-time interactive humanoid video generation. Built upon a MMDiT architecture, FlowAct-R1 enables the streaming synthesis of video with arbitrary durations while maintaining low-latency responsiveness. We introduce a chunkwise diffusion forcing strategy, complemented by a novel self-forcing variant, to alleviate error accumulation and ensure long-term temporal consistency during continuous interaction. By leveraging efficient distillation and system-level optimizations, our framework achieves a stable 25fps at 480p resolution with a time-to-first-frame (TTFF) of only around 1.5 seconds. The proposed method provides holistic and fine-grained full-body control, enabling the agent to transition naturally between diverse behavioral states in interactive scenarios. Experimental results demonstrate that FlowAct-R1 achieves exceptional behavioral vividness and perceptual realism, while maintaining robust generalization across diverse character styles.
Paper Structure (10 sections, 3 figures, 1 table)

This paper contains 10 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Approach
  3. Overall Pipeline
  4. Training Procedure and Framework Design
  5. Multimodal Fusion
  6. Model Acceleration
  7. Inference Optimization
  8. Multimodal Action Planning
  9. Experiments
  10. Conclusion

Figures (3)

  • Figure 1: We present FlowAct-R1, a novel framework that enables lifelike, responsive, and high-fidelity humanoid video generation for seamless real-time interaction.
  • Figure 2: Overview of the FlowAct-R1 framework. It consists of training and inference stages: training integrates converting base full-attention DiT to streaming AR model via autoregressive adaptation, joint audio-motion finetuning for better lip-sync and body motion, multi-stage diffusion distillation; inference adopts a structured memory bank (Reference/Long/Short-term Memory, Denoising Stream) with chunkwise autoregressive generation and memory refinement. Complemented by system-level optimizations, it achieves 25fps real-time 480p video generation (TTFF 1.5s) with vivid behavioral transitions.
  • Figure 3: Comparisons with KlingAvatar 2.0 klingteam2025klingavatar20technicalreport, LiveAvatar huang2025liveavatarstreamingrealtime, and Omnihuman-1.5 jiang2025omnihuman15instillingactivemind via a user study using the GSB (good-same-bad) metric. The $\bfseries\sffamily{\textcolor{rgb(255,127,14)}{orange segments}}$ indicate the percentage of user votes favoring FlowAct-R1 over other methods. Video demos are shown in our project page.