Think-Then-React: Towards Unconstrained Human Action-to-Reaction Generation

TL;DR

This work tackles online, unconstrained action-to-reaction generation in multi-person scenes, where reactions must be produced without prompts and with robust handling of both egocentric pose and absolute space. It introduces Think-Then-React (TTR), an LLM-based framework that first reasons about action intents and then generates reactions conditioned on the inferred semantics, aided by a unified space-pose motion tokenizer. The approach combines a decoupled egocentric pose tokenizer and an absolute space tokenizer, plus a multi-task pre-training regimen and causal fine-tuning that include a periodic re-thinking mechanism to curb error accumulation. Empirically, TTR surpasses state-of-the-art baselines by substantial margins on the Inter-X dataset, notably achieving an FID improvement from 3.988 to 1.942, and demonstrates enhanced reasoning, efficiency, and real-time capability for action-to-reaction generation with multi-person interactions.

Abstract

Modeling human-like action-to-reaction generation has significant real-world applications, like human-robot interaction and games. Despite recent advancements in single-person motion generation, it is still challenging to well handle action-to-reaction generation, due to the difficulty of directly predicting reaction from action sequence without prompts, and the absence of a unified representation that effectively encodes multi-person motion. To address these challenges, we introduce Think-Then-React (TTR), a large language-model-based framework designed to generate human-like reactions. First, with our fine-grained multimodal training strategy, TTR is capable to unify two processes during inference: a thinking process that explicitly infers action intentions and reasons corresponding reaction description, which serve as semantic prompts, and a reacting process that predicts reactions based on input action and the inferred semantic prompts. Second, to effectively represent multi-person motion in language models, we propose a unified motion tokenizer by decoupling egocentric pose and absolute space features, which effectively represents action and reaction motion with same encoding. Extensive experiments demonstrate that TTR outperforms existing baselines, achieving significant improvements in evaluation metrics, such as reducing FID from 3.988 to 1.942.

Figures (7)

• Figure 1: Given a human action as input, our Think-Then-React model first thinks by generating an action description and reasons out a reaction prompt. It then reacts to the action based on the results of this thinking process. TTR reacts in a real-time manner at every timestep and periodically re-thinks at specific interval (every two timesteps in the illustration) to mitigate accumulated errors.
• Figure 2: Illustration of our decoupled tokenizer and the plain tokenizer.
• Figure 3: Visualization of a person's motion sequences in Inter-X dataset and HumanML3D dataset.
• Figure 4: Visualized cases of our predicted reactions (in green) to input action (in blue) and corresponding thinking results. We also provide a failure case in figure (d), where TTR misunderstands the input action as "wrestling", which should be "embracing".
• Figure 5: Impact of input action FPS to summed ranking score differences.