Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NanoHTNet: Nano Human Topology Network for Efficient 3D Human Pose Estimation

Jialun Cai, Mengyuan Liu, Hong Liu, Shuheng Zhou, Wenhao Li

TL;DR

3D human pose estimation on edge devices is hindered by high computational demands. The paper introduces NanoHTNet, a compact dual-stream encoder with Hierarchical Mixers and Efficient Temporal-Spatial Tokenization (ETST) to capture explicit skeletal priors while reducing complexity, complemented by PoseCLR, a cross-view contrastive pre-training strategy for learning implicit topology priors. Key innovations include a three-level spatial topology (LJC, IPC, GBI), temporal multi-scale mixers (IME, GCP), and frequency-domain data compression, which together enable real-time inference on Jetson Nano and strong generalization to unseen environments. The combination yields a practical edge-enabled 3D HPE pipeline with improved initialization for various backbones and publicly available code and models.

Abstract

The widespread application of 3D human pose estimation (HPE) is limited by resource-constrained edge devices, requiring more efficient models. A key approach to enhancing efficiency involves designing networks based on the structural characteristics of input data. However, effectively utilizing the structural priors in human skeletal inputs remains challenging. To address this, we leverage both explicit and implicit spatio-temporal priors of the human body through innovative model design and a pre-training proxy task. First, we propose a Nano Human Topology Network (NanoHTNet), a tiny 3D HPE network with stacked Hierarchical Mixers to capture explicit features. Specifically, the spatial Hierarchical Mixer efficiently learns the human physical topology across multiple semantic levels, while the temporal Hierarchical Mixer with discrete cosine transform and low-pass filtering captures local instantaneous movements and global action coherence. Moreover, Efficient Temporal-Spatial Tokenization (ETST) is introduced to enhance spatio-temporal interaction and reduce computational complexity significantly. Second, PoseCLR is proposed as a general pre-training method based on contrastive learning for 3D HPE, aimed at extracting implicit representations of human topology. By aligning 2D poses from diverse viewpoints in the proxy task, PoseCLR aids 3D HPE encoders like NanoHTNet in more effectively capturing the high-dimensional features of the human body, leading to further performance improvements. Extensive experiments verify that NanoHTNet with PoseCLR outperforms other state-of-the-art methods in efficiency, making it ideal for deployment on edge devices like the Jetson Nano. Code and models are available at https://github.com/vefalun/NanoHTNet.

NanoHTNet: Nano Human Topology Network for Efficient 3D Human Pose Estimation

TL;DR

3D human pose estimation on edge devices is hindered by high computational demands. The paper introduces NanoHTNet, a compact dual-stream encoder with Hierarchical Mixers and Efficient Temporal-Spatial Tokenization (ETST) to capture explicit skeletal priors while reducing complexity, complemented by PoseCLR, a cross-view contrastive pre-training strategy for learning implicit topology priors. Key innovations include a three-level spatial topology (LJC, IPC, GBI), temporal multi-scale mixers (IME, GCP), and frequency-domain data compression, which together enable real-time inference on Jetson Nano and strong generalization to unseen environments. The combination yields a practical edge-enabled 3D HPE pipeline with improved initialization for various backbones and publicly available code and models.

Abstract

The widespread application of 3D human pose estimation (HPE) is limited by resource-constrained edge devices, requiring more efficient models. A key approach to enhancing efficiency involves designing networks based on the structural characteristics of input data. However, effectively utilizing the structural priors in human skeletal inputs remains challenging. To address this, we leverage both explicit and implicit spatio-temporal priors of the human body through innovative model design and a pre-training proxy task. First, we propose a Nano Human Topology Network (NanoHTNet), a tiny 3D HPE network with stacked Hierarchical Mixers to capture explicit features. Specifically, the spatial Hierarchical Mixer efficiently learns the human physical topology across multiple semantic levels, while the temporal Hierarchical Mixer with discrete cosine transform and low-pass filtering captures local instantaneous movements and global action coherence. Moreover, Efficient Temporal-Spatial Tokenization (ETST) is introduced to enhance spatio-temporal interaction and reduce computational complexity significantly. Second, PoseCLR is proposed as a general pre-training method based on contrastive learning for 3D HPE, aimed at extracting implicit representations of human topology. By aligning 2D poses from diverse viewpoints in the proxy task, PoseCLR aids 3D HPE encoders like NanoHTNet in more effectively capturing the high-dimensional features of the human body, leading to further performance improvements. Extensive experiments verify that NanoHTNet with PoseCLR outperforms other state-of-the-art methods in efficiency, making it ideal for deployment on edge devices like the Jetson Nano. Code and models are available at https://github.com/vefalun/NanoHTNet.
Paper Structure (16 sections, 8 equations, 10 figures, 9 tables)

This paper contains 16 sections, 8 equations, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficient 3D HPE
  4. Human Topology Modeling in 3D HPE
  5. Self-Supervised Learning in 3D HPE
  6. Methods
  7. Encoding Network Structure
  8. Hierarchical Mixers
  9. PoseCLR
  10. Experiments
  11. Datasets and Evaluation
  12. Implementation Details
  13. Comparison with State-of-the-art Results
  14. Ablation Study
  15. Visualization
  16. ...and 1 more sections

Figures (10)

  • Figure 1: Comparison of performance (MPJPE) and speed (FPS) between the proposed NanoHTNet and SOTA methods on the Jetson Nano, a compact and power-efficient AI device. NanoHTNet significantly excels in the speed-accuracy trade-off, enabling real-time inference for 3D HPE in edge AI.
  • Figure 2: Spatio-temporal priors of the human skeleton: (a) Spatial hierarchical structure; (b) Temporal multi-scale kinematics; (c) Cross-view spatio-temporal topological consistency.
  • Figure 3: Overview of our encoding network NanoHTNet. A two-stream structure is designed for independently extracting the spatial-domain physical topology and the temporal-domain kinematic topology of the human body. $T$ denotes input frames, while $T_k$ represents the reduced frames after applying DCT and LF. $C$ is the main channel dimension after patch embedding, and $C_l$ is the channel dimension after MLP networks $T\_FCN$ and $S\_FCN$.
  • Figure 4: Stacked Hierarchical Mixers are key in our encoder. Each Spatial Hierarchical Mixer consists of Local Joint-level Connection (LJC), Intra-Part Constraint (IPC), and Global Body-level Interaction (GBI). Each Temporal Hierarchical Mixer effectively captures temporal features from local to global extents by Instantaneous Motion Extraction (IME) and Global Coherence Perception (GCP).
  • Figure 5: (a) Specific estimation error distribution caused by neglect of part-level features; (b) Structure of Intra-Part Constraint (IPC) module.
  • ...and 5 more figures