FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance

TL;DR

FinePhys tackles the challenge of generating physically plausible, fine-grained human action videos from monocular input by integrating physics into skeletal-guided diffusion. It introduces a physics-informed diffusion framework that combines online 2D pose estimation, in-context learning for 2D-to-3D pose lifting, and a differentiable PhysNet module enforcing the Euler-Lagrange dynamics $M(q)\ddot{q} = J(q,\dot{q}) - C(q,\dot{q})$ to produce physics-refined 3D skeletons, which are fused with data-driven poses to guide the diffusion process via multi-scale skeletal heatmaps. The approach leverages observational, inductive, and learning biases, with stage-wise losses that promote physical consistency and efficient parameterization via LoRA in the 3D-UNet. Evaluations on FineGym FX-JUMP, FX-TURN, and FX-SALTO show substantial improvements over baselines in both quantitative metrics (including the enhanced CLIP-SIM$^*$ and user studies) and qualitative analyses, yielding more natural and biomechanically plausible actions. This work advances physically grounded video synthesis for complex, high-deformation human motions and offers a pathway toward reliable fine-grained action generation for animation and robotics.

Abstract

Despite significant advances in video generation, synthesizing physically plausible human actions remains a persistent challenge, particularly in modeling fine-grained semantics and complex temporal dynamics. For instance, generating gymnastics routines such as "switch leap with 0.5 turn" poses substantial difficulties for current methods, often yielding unsatisfactory results. To bridge this gap, we propose FinePhys, a Fine-grained human action generation framework that incorporates Physics to obtain effective skeletal guidance. Specifically, FinePhys first estimates 2D poses in an online manner and then performs 2D-to-3D dimension lifting via in-context learning. To mitigate the instability and limited interpretability of purely data-driven 3D poses, we further introduce a physics-based motion re-estimation module governed by Euler-Lagrange equations, calculating joint accelerations via bidirectional temporal updating. The physically predicted 3D poses are then fused with data-driven ones, offering multi-scale 2D heatmap guidance for the diffusion process. Evaluated on three fine-grained action subsets from FineGym (FX-JUMP, FX-TURN, and FX-SALTO), FinePhys significantly outperforms competitive baselines. Comprehensive qualitative results further demonstrate FinePhys's ability to generate more natural and plausible fine-grained human actions.

Figures (19)

• Figure 1: Video generation results for fine-grained human action "split leap with 1 turn". Our FinePhys demonstrates superior performance in generating physically plausible fine-grained human actions, while SOTA methods exhibit significant issues, including severe temporal inconsistencies guo2023animatediff, noticeable limb distortions ma2024follow, and character anomalies chen2023videocrafter1.
• Figure 2: Overview of Finephys. FinePhys addresses the challenging task of generating fine-grained human action videos by explicitly incorporating physical equations exploiting pose modality. The pipeline begins with online extracting 2D poses, then transforms them into 3D using an in-context learning module, achieving the data-driven 3D skeleton sequence $S^{3D}_{dd}$. To incorporate the physical laws of motion, we introduce a Phys-Net module to re-estimate the 3D positions of each human joint by accounting for second-order temporal variations (i.e., accelerations) in both forward and reverse directions, yielding physically predicted 3D poses $S^{3D}_{pp}$. Subsequently, $S^{3D}_{dd}$ and $S^{3D}_{pp}$ are fused, projected back into 2D space, encoded into multi-scale latent maps, and integrated into 3D-UNets to guide the denoising process.
• Figure 3: The PhysNet Module. Given the input $S_{dd}^{3D}$, PhysNet leverages both global and local temporal dynamics in a bi-directional manner to estimate the terms of the Euler-Lagrange equations. By integrating with an ODE solver, the module can predict future and past states, thereby enhancing the original $S_{dd}^{3D}$ across both temporal directions and producing physically predicted 3D sequences, denoted as $S_{pp}^{3D}$.
• Figure 4: Original CLIP-SIM metrics fail to evaluate the generated results (e.g., T2I-Zero produces entirely irrelevant outputs yet achieves the highest smooth score according to the original CLIP-SIM. In contrast, our enhanced CLIP-SIM* provides a more reliable evaluation that better aligns with human judgment.
• Figure 5: Qualitative Results. Compared to other baselines, FinePhys demonstrates superior performance in understanding complex, fine-grained semantics, maintaining biomechanical consistency, and adhering to physical principles.