Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer
Tasmiah Haque, Srinjoy Das
TL;DR
The paper tackles the need for diverse and temporally coherent forecasts in real-time keypoint-based video motion transfer, where deterministic GRU-NF can underrepresent multimodal futures. It introduces GRU-SNF, an inference-time refinement that inserts a small number of MCMC steps between normalizing-flow layers of a pretrained GRU-NF, guided by an energy that blends a Gaussian prior with GRU-driven predictions, without retraining. Across VoxCeleb and BAIR, GRU-SNF yields improved diversity (lower energy distance on keypoints and higher APD) while maintaining fidelity, with gains that grow at longer horizons and only modest latency. This approach demonstrates the practical viability of lightweight stochastic refinement for bandwidth-efficient, multimodal time-series forecasting in real-time systems and suggests broader applicability to flow-based sequence models.
Abstract
Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision making under uncertainty. To improve the diversity of such sequential forecasts we propose a novel inference-time refinement technique that combines Gated Recurrent Unit-Normalizing Flows (GRU-NF) with stochastic sampling methods. While GRU-NF can capture multimodal distributions through its integration of normalizing flows within a temporal forecasting framework, its deterministic transformation structure can limit expressivity. To address this, inspired by Stochastic Normalizing Flows (SNF), we introduce Markov Chain Monte Carlo (MCMC) steps during GRU-NF inference, enabling the model to explore a richer output space and better approximate the true data distribution without retraining. We validate our approach in a keypoint-based video motion transfer pipeline, where capturing temporally coherent and perceptually diverse future trajectories is essential for realistic samples and low bandwidth communication. Experiments show that our inference framework, Gated Recurrent Unit- Stochastic Normalizing Flows (GRU-SNF) outperforms GRU-NF in generating diverse outputs without sacrificing accuracy, even under longer prediction horizons. By injecting stochasticity during inference, our approach captures multimodal behavior more effectively. These results highlight the potential of integrating stochastic dynamics with flow-based sequence models for generative time series forecasting.