Efficient 3D Full-Body Motion Generation from Sparse Tracking Inputs with Temporal Windows
Georgios Fotios Angelis, Savas Ozkan, Sinan Mutlu, Paul Wisbey, Anastasios Drosou, Mete Ozay
TL;DR
This work tackles real-time 3D full-body motion generation from sparse upper-body sensors for VR/AR by introducing a temporal-windowed MLP (TW-MLP). The method partitions long input sequences into short windows of length $T$ and fuses past context through latent representations from $K$ past windows, improving accuracy while dramatically reducing compute to enable on-device deployment. By combining an uncertainty-based multi-task loss with a lightweight MLP backbone and latent past-context fusion, the approach achieves competitive reconstruction quality with only $\text{FLOPs}$ of about $0.19$G and real-time performance on devices like Meta Quest-3 (around 72 FPS). Experiments on AMASS corroborate substantial gains in jitter reduction and efficiency, with ablations confirming favorable choices for $T$, $K$, and concatenation strategies. Overall, the method offers a practical, high-accuracy, resource-efficient solution for immersive AR/VR applications on constrained hardware.
Abstract
To have a seamless user experience on immersive AR/VR applications, the importance of efficient and effective Neural Network (NN) models is undeniable, since missing body parts that cannot be captured by limited sensors should be generated using these models for a complete 3D full-body reconstruction in virtual environment. However, the state-of-the-art NN-models are typically computational expensive and they leverage longer sequences of sparse tracking inputs to generate full-body movements by capturing temporal context. Inevitably, longer sequences increase the computation overhead and introduce noise in longer temporal dependencies that adversely affect the generation performance. In this paper, we propose a novel Multi-Layer Perceptron (MLP)-based method that enhances the overall performance while balancing the computational cost and memory overhead for efficient 3D full-body generation. Precisely, we introduce a NN-mechanism that divides the longer sequence of inputs into smaller temporal windows. Later, the current motion is merged with the information from these windows through latent representations to utilize the past context for the generation. Our experiments demonstrate that generation accuracy of our method with this NN-mechanism is significantly improved compared to the state-of-the-art methods while greatly reducing computational costs and memory overhead, making our method suitable for resource-constrained devices.