Beyond Motion Pattern: An Empirical Study of Physical Forces for Human Motion Understanding

TL;DR

The paper addresses the limitations of appearance- and kinematics-based human motion understanding by introducing physically inferred forces, specifically joint torques, as a new modality. It estimates torques from monocular video and encodes them with a Force Network, then fuses these signals with visual/kinematic cues across gait recognition, action recognition, and fine-grained video captioning. Across eight benchmarks, force dynamics yield consistent gains, particularly for high-exertion actions and under appearance or viewpoint changes, and they enhance temporal grounding and semantic richness in captions. The findings demonstrate that physics-informed cues provide a general, complementary source of information that improves robustness and descriptive quality in real-world motion understanding systems.

Abstract

Human motion understanding has advanced rapidly through vision-based progress in recognition, tracking, and captioning. However, most existing methods overlook physical cues such as joint actuation forces that are fundamental in biomechanics. This gap motivates our study: if and when do physically inferred forces enhance motion understanding? By incorporating forces into established motion understanding pipelines, we systematically evaluate their impact across baseline models on 3 major tasks: gait recognition, action recognition, and fine-grained video captioning. Across 8 benchmarks, incorporating forces yields consistent performance gains; for example, on CASIA-B, Rank-1 gait recognition accuracy improved from 89.52% to 90.39% (+0.87), with larger gain observed under challenging conditions: +2.7% when wearing a coat and +3.0% at the side view. On Gait3D, performance also increases from 46.0% to 47.3% (+1.3). In action recognition, CTR-GCN achieved +2.00% on Penn Action, while high-exertion classes like punching/slapping improved by +6.96%. Even in video captioning, Qwen2.5-VL's ROUGE-L score rose from 0.310 to 0.339 (+0.029), indicating that physics-inferred forces enhance temporal grounding and semantic richness. These results demonstrate that force cues can substantially complement visual and kinematic features under dynamic, occluded, or appearance-varying conditions.

Figures (8)

• Figure 1: Visualization of RGB frames and their corresponding motion forces from multiple datasets used in gait recognition, action recognition, and video captioning tasks. The magnitude of joint actuations is encoded by color at each body part, with lighter colors indicating higher magnitudes.
• Figure 2: Overview of our framework. Force estimated from video via physics-informed model are fused with conventional visual modalities (RGB, pose, silhouette) through feature-level or decision-level fusion, and then used for task-specific adaptation.
• Figure 3: Comparison of Qwen2.5-VL on video captioning task. We compare captions generated by the baseline model (Without Force) and our force-augmented model (With Force). The green-highlighted words indicate phrases that explicitly capture biomechanical effort, while the red highlights mark imprecise phrases produced by the baseline model.
• Figure 4: Joint-level force ablation on the NTU-RGB+D 60 dataset. Accuracy decreases are measured when masking each joint's force feature from the action recognition network. The model is most sensitive to forces around the shoulders, upper head, and knees, highlighting that upper-body and supporting-limb dynamics play key roles in distinguishing actions.
• Figure 5: Joint-level force ablation on the CASIA-B gait dataset. Accuracy decreases are measured when masking each joint's force feature from gait-recognition network. The greatest degradation occurs at the ankles, spine base and hips, showing that lower-body propulsion and balance forces dominate gait identity cues, while upper-body joints contribute less.