QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture

TL;DR

QuaMo addresses jittery motion in monocular 3D human motion capture by replacing Euler-based joint representations with quaternion-based joints and solving a quaternion differential equation (QDE) under a unit-sphere constraint $\mathcal{S}^3$. It combines an online state-space formulation with a meta-PD controller that includes a novel acceleration enhancement to adapt to rapid pose changes, yielding continuous, plausible 3D kinematics from single-frame inputs. The approach uses InitNet and ControlNet to predict initial conditions and PD gains, achieving state-of-the-art results on Human3.6M, Fit3D, SportsPose, and AIST among online methods while maintaining real-time performance. This work demonstrates the value of exact quaternion integration and adaptive control for robust, online motion capture, with strong potential for practical vision-based kinematics and biomechanics applications.

Abstract

Vision-based 3D human motion capture from videos remains a challenge in computer vision. Traditional 3D pose estimation approaches often ignore the temporal consistency between frames, causing implausible and jittery motion. The emerging field of kinematics-based 3D motion capture addresses these issues by estimating the temporal transitioning between poses instead. A major drawback in current kinematics approaches is their reliance on Euler angles. Despite their simplicity, Euler angles suffer from discontinuity that leads to unstable motion reconstructions, especially in online settings where trajectory refinement is unavailable. Contrarily, quaternions have no discontinuity and can produce continuous transitions between poses. In this paper, we propose QuaMo, a novel Quaternion Motions method using quaternion differential equations (QDE) for human kinematics capture. We utilize the state-space model, an effective system for describing real-time kinematics estimations, with quaternion state and the QDE describing quaternion velocity. The corresponding angular acceleration is computed from a meta-PD controller with a novel acceleration enhancement that adaptively regulates the control signals as the human quickly changes to a new pose. Unlike previous work, our QDE is solved under the quaternion unit-sphere constraint that results in more accurate estimations. Experimental results show that our novel formulation of the QDE with acceleration enhancement accurately estimates 3D human kinematics with no discontinuity and minimal implausibilities. QuaMo outperforms comparable state-of-the-art methods on multiple datasets, namely Human3.6M, Fit3D, SportsPose and AIST. The code is available at https://github.com/cuongle1206/QuaMo

Figures (8)

• Figure 1: We present QuaMo, a novel online 3D human kinematics capture approach based on Quaternion Motions (pink), modeled via a meta-PD algorithm with acceleration enhancement. Given a vision-based 3D pose estimation as prior, QuaMo predicts plausible and accurate motions.
• Figure 2: QuaMo consists of two differentiable equations: ODE for angular velocity $\omega$ and QDE for quaternion pose $q$. The updated $\omega_{t+\Delta t}$ is computed via a data-driven meta-PD controller with the additional adaptive signals from our novel second-order acceleration enhancement and Euler integration. Given $\omega_{t+\Delta t}$, the next human pose $q_{t+\Delta t}$ is updated by solving the QDE with the Hamilton quaternion product. The human body mesh $m_{t+\Delta t}$ and the corresponding keypoints $p_{t+\Delta t}$ are retrieved by applying a linear transformation with the SMPL skinned model from pavlakos2019_smplx, taking the pose $q_{t+\Delta t}$ and shape parameter $\beta$ as inputs.
• Figure 3: Qualitative results on three datasets: Fit3D (left), SportsPose (middle), AIST (right). QuaMo's predictions are shown in blue, the input (from TRACE or HMR2.0) in green, and ground truth keypoints in red for reference. The start frame has lower transparency. The reconstructed motions from QuaMo have significantly lower jitter and higher accuracy along the optical axis.
• Figure 4: An example of motion reconstruction when a discontinuity occurs in the root joint rotation for different rotation representations. Blue means low and orange high MPJPE. The transparency corresponds to the time steps in the sequence. The model attempts to compensate for the discontinuity by rotating along the different rotation axes for all representations, except our quaternions.
• Figure 5: The architecture of ControlNet. The inputs are concatenated and linearly mapped to an embedding vector of 512 dimensions. The control parameters for the PD controller is predicted from the embedding via linear mappings, sigmoid functions and scaling.