Autoregressive Flow Matching for Motion Prediction

TL;DR

Autoregressive Flow Matching (ARFM) presents a scalable, probabilistic framework for predicting future motion across diverse video domains by modeling future point tracks rather than full pixel data. It combines a multimodal feature fusion encoder with a two-stage autoregressive flow predictor, enabling online horizon updates and conditioning on language and frames. Through a pseudo-labeling data pipeline and extensive zero-shot evaluation on human and robotics benchmarks, ARFM demonstrates improved track prediction and enhanced downstream task performance in interaction synthesis and robotic control. The approach highlights the practicality of using track-level representations and large unlabeled video data to build a generalized motion-prediction foundation model with real-time inference capabilities.

Abstract

Motion prediction has been studied in different contexts with models trained on narrow distributions and applied to downstream tasks in human motion prediction and robotics. Simultaneously, recent efforts in scaling video prediction have demonstrated impressive visual realism, yet they struggle to accurately model complex motions despite massive scale. Inspired by the scaling of video generation, we develop autoregressive flow matching (ARFM), a new method for probabilistic modeling of sequential continuous data and train it on diverse video datasets to generate future point track locations over long horizons. To evaluate our model, we develop benchmarks for evaluating the ability of motion prediction models to predict human and robot motion. Our model is able to predict complex motions, and we demonstrate that conditioning robot action prediction and human motion prediction on predicted future tracks can significantly improve downstream task performance. Code and models publicly available at: https://github.com/Johnathan-Xie/arfm-motion-prediction.

Figures (6)

• Figure 1: Generalized motion prediction via autoregressive flow matching. To train our method (top), we extract pseudo label tracks using an off-the-shelf point tracker, then train an autoregressive flow matching model to predict tracks. To apply our method to downstream tasks (bottom), we generate future point tracks conditioned on provided video frames and an optional language prompt. Then, we extract features from these tracks which are used by a task specific model to accomplish the task.
• Figure 2: Overview of feature fusion module. Our feature fusion module which takes video frames, past point tracks, and an optional natural language caption. These multi-modal inputs are aggregated into a single feature vector for each future track prediction location.
• Figure 3: Overview of training flow matching predictor. Using each of the track level features and relative shift inputs, the model attends spatially to denoise temporal track differences which are used during inference time for sampling.
• Figure 4: Horizon update overview. To efficiently update a predicted horizon, we first obtain a newly tracked point using an online point tracker. Then, we apply noise to previously predicts tracks followed by denoising from an intermediate timestep to update the tracks while retaining continuity. Finally, we fully denoise a new point at the final timestep.
• Figure 5: Motion generation visualizations. In the top row, we show a visualization on the evaluation set of UCF-101. We find our model can correctly predict the right fencer lunging toward the left one as well as predict the proper camera motion of the background moving right as the camera pans left to follow the motion of the fencers. In the bottom row, we show a visualization on the CALVIN robotics benchmark. Here, we see the model can properly guide the robot arm to the correctly colored block to complete the task. Furthermore, through our updating horizon prediction, the model is capable of differentiating between the block colors and correctly predicts when to begin grasping the block. Additional results are shown in the appendix.