EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation

TL;DR

This work tackles the challenge of generating temporally coherent and spatially grounded 3D human reactions from egocentric video in real time. By constructing the HRD dataset with spatially aligned egocentric video–motion pairs and introducing EgoReAct, a two-stage autoregressive framework using a Motion VQ-VAE and a Transformer, the authors achieve causal, 3D-consistent reaction synthesis guided by rich ego-perception cues including visual semantics, depth, and head dynamics. Ablation and ablation-style experiments show the benefits of incorporating metric depth and ego dynamics for grounding, while comparisons against state-of-the-art baselines demonstrate superior realism, spatial alignment, and online efficiency. The approach has practical implications for embodied AI and responsive human–robot interaction, with real-world deployment validating robustness to dynamic scenes.

Abstract

Humans exhibit adaptive, context-sensitive responses to egocentric visual input. However, faithfully modeling such reactions from egocentric video remains challenging due to the dual requirements of strictly causal generation and precise 3D spatial alignment. To tackle this problem, we first construct the Human Reaction Dataset (HRD) to address data scarcity and misalignment by building a spatially aligned egocentric video-reaction dataset, as existing datasets (e.g., ViMo) suffer from significant spatial inconsistency between the egocentric video and reaction motion, e.g., dynamically moving motions are always paired with fixed-camera videos. Leveraging HRD, we present EgoReAct, the first autoregressive framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time. We first compress the reaction motion into a compact yet expressive latent space via a Vector Quantised-Variational AutoEncoder and then train a Generative Pre-trained Transformer for reaction generation from the visual input. EgoReAct incorporates 3D dynamic features, i.e., metric depth, and head dynamics during the generation, which effectively enhance spatial grounding. Extensive experiments demonstrate that EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods, while maintaining strict causality during generation. We will release code, models, and data upon acceptance.

Figures (9)

• Figure 1: EgoReAct takes streaming egocentric video as input and synthesizes spatially grounded, realistic human reaction motions in real time, enabling responsive full-body behaviors that are tightly coupled with the ongoing egocentric observations.
• Figure 2: The automated pipeline for generating the Spatially Aligned Human Reaction Dataset (HRD). Given a scene caption, we first employ LLMs to produce video and motion prompts, then generate egocentric videos and reaction motions through text-driven generation, followed by spatial alignment via camera trajectory control.
• Figure 3: Dataset Distribution of the HRD dataset. The dataset consists of three main categories: human-human (blue), animal-human (green), and scene-human (red) interactions.
• Figure 4: Pipeline of EgoReAct. We first learn a Motion VQ-VAE to discretize continuous 3D motions into compact token sequences. Building on this representation, EgoReAct takes streaming egocentric RGB frames as input, estimates their depth, and encodes the image, depth, and head dynamics cues to form ego-perception features, which guide an Autoregressive Transformer to sequentially generate spatially aligned and temporally causal reaction motions.
• Figure 5: Comparison between ViMo yu2025hero and our Spatially Aligned Human Reaction Dataset (HRD). The left side shows the ground-truth reaction motion. On the right, the top row presents the egocentric video from ViMo, while the bottom row shows the video from our HRD. Our dataset provides significantly more accurate spatial alignment between the egocentric video and the reaction motion.