The Last Mile to Production Readiness: Physics-Based Motion Refinement for Video-Based Capture

TL;DR

Vision-based motion capture yields visually compelling data but requires physics-based cleanup for production realism. The authors introduce a per-sequence physics-based motion refinement framework powered by deep reinforcement learning, addressing artifacts such as penetration, floating/weightedness, phantom contact, and foot sliding, with support for single- and multi-character sequences. Key contributions include a penetration-depth–driven artifact severity estimator $ ho_t$, a PD-base imitation policy with $q_b = \text{SLERP}(q_t, \hat{q}_{t+1}, \alpha)$, adaptive termination using $d_p^j$ scaled by $\rho_t$, and a MAPPO-based multi-character cleanup plus a user-guided workflow. The approach reduces manual cleanup, improves physical realism, and integrates into animator pipelines, providing a path toward production-ready motion assets from vision-based capture. Overall, the framework demonstrates substantial cleanup across multiple artifact types and offers practical mechanisms for production use and future generalization.

Abstract

High-quality motion data underpins games, film, XR, and robotics. Vision-based motion capture tools have made significant progress, offering accessible and visually convincing results, yet often fall short in the final stretch -- the last mile -- when it comes to physical realism and production readiness, due to various artifacts introduced during capture. In this paper, we summarize key issues through case studies and feedback from professional animators to set a stepping stone for future research in motion cleanup. We then present a physics-based motion refinement framework to bridge the gap, with the goal of reducing labor-intensive manual cleanup and enhancing visual quality and physical realism. Our framework supports both single- and multi-character sequences and can be integrated into animator workflows for further refinement, such as stylizing motions via keyframe editing.

Figures (5)

• Figure 1: A sharp stop motion showing over-smoothed hip (root) trajectory in the raw input (pink), which lacks a sense of weight. The refined motion (teal) from our method adds realistic root movement.
• Figure 2: Over-smoothed hip trajectory during locomotion. The plot compares hip height from raw motion (pink) and our physical reconstruction (teal).
• Figure 3: Penetration depth in the motion clip where two characters run into each other. The peaks in the penetration depth are highlighted with blue circles.
• Figure 4: Foot sliding example. The pink character shows sliding artifacts, while the teal version from our method maintains proper foot contact. Blue circle marks a planted right foot, which should remain stationary.
• Figure 5: Pink character represents the raw motion, and the teal one represents the motion generated by our framework.