EA-RAS: Towards Efficient and Accurate End-to-End Reconstruction of Anatomical Skeleton

TL;DR

The EA-RAS (Towards Efficient and Accurate End-to-End Reconstruction of Anatomical Skeleton), a single-stage, lightweight, and plug-and-play anatomical skeleton estimator that can provide real-time, accurate anatomically realistic skeletons with arbitrary pose using only a single RGB image input is proposed.

Abstract

Efficient, accurate and low-cost estimation of human skeletal information is crucial for a range of applications such as biology education and human-computer interaction. However, current simple skeleton models, which are typically based on 2D-3D joint points, fall short in terms of anatomical fidelity, restricting their utility in fields. On the other hand, more complex models while anatomically precise, are hindered by sophisticate multi-stage processing and the need for extra data like skin meshes, making them unsuitable for real-time applications. To this end, we propose the EA-RAS (Towards Efficient and Accurate End-to-End Reconstruction of Anatomical Skeleton), a single-stage, lightweight, and plug-and-play anatomical skeleton estimator that can provide real-time, accurate anatomically realistic skeletons with arbitrary pose using only a single RGB image input. Additionally, EA-RAS estimates the conventional human-mesh model explicitly, which not only enhances the functionality but also leverages the outside skin information by integrating features into the inside skeleton modeling process. In this work, we also develop a progressive training strategy and integrated it with an enhanced optimization process, enabling the network to obtain initial weights using only a small skin dataset and achieve self-supervision in skeleton reconstruction. Besides, we also provide an optional lightweight post-processing optimization strategy to further improve accuracy for scenarios that prioritize precision over real-time processing. The experiments demonstrated that our regression method is over 800 times faster than existing methods, meeting real-time requirements. Additionally, the post-processing optimization strategy provided can enhance reconstruction accuracy by over 50% and achieve a speed increase of more than 7 times.

Figures (15)

• Figure 1: Our method completes both human skin and anatomical skeleton reconstructions. The model achieves good inference results on 3DPW von2018recoveringand the lower resolution LSP johnson2010clustered dataset.
• Figure 2: An overview of the entire proposed method. Our method employs an iterative optimization approach to achieve self-supervision of regression results. The regression network is augmented with our designed HTS feature, and the network predicts the parameters of human SMPL SMPL:2015 and skeleton model Keller:CVPR:2022. Skeleton and human body features are integrated and complemented through the OHTS in the process of optimization supervision.
• Figure 3: An overview of the OHTS. Keypoints serve as implicit supervision in the regression process. The results of the optimization phase provide explicit supervision of the regression results. Anatomical joint reg and KP helps to get the keypoints and shape of skeleton. Colored regions represent vertices affecting skeletal keypoints, while the heatmap indicates the correlation between body shape and skeletal parameterization.
• Figure 4: Parameter Decomposition: The paramater is optimized simultaneously to control the posture of the skeleton shapes. Match points (MP) and control points (CP) can control joint and position to fit topological cluster. The entire skeleton's topology cluster can be optimized using the corresponding control points. The cuboid represents the skeletal coordinate system, linking control points to form skeletal morphology. Cost Control showcases how our optimization process controls the skeleton by maintaining the relationship between vertices and control points (CP). It also ensures that the skeletal and human body mesh points align through cross-sections and maintains the spacing relationship between mesh points (MP) during optimization.
• Figure 5: The loss of Rodrigues and Quaternion