Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds

Abstract

Understanding humans from LiDAR point clouds is one of the most critical tasks in autonomous driving due to its close relationships with pedestrian safety, yet it remains challenging in the presence of diverse human-object interactions and cluttered backgrounds. Nevertheless, existing methods largely overlook the potential of leveraging human-object interactions to build robust 3D human pose estimation frameworks. There are two major challenges that motivate the incorporation of human-object interaction. First, human-object interactions introduce spatial ambiguity between human and object points, which often leads to erroneous 3D human keypoint predictions in interaction regions. Second, there exists severe class imbalance in the number of points between interacting and non-interacting body parts, with the interaction-frequent regions such as hand and foot being sparsely observed in LiDAR data. To address these challenges, we propose a Human-Object Interaction Learning (HOIL) framework for robust 3D human pose estimation from LiDAR point clouds. To mitigate the spatial ambiguity issue, we present human-object interaction-aware contrastive learning (HOICL) that effectively enhances feature discrimination between human and object points, particularly in interaction regions. To alleviate the class imbalance issue, we introduce contact-aware part-guided pooling (CPPool) that adaptively reallocates representational capacity by compressing overrepresented points while preserving informative points from interacting body parts. In addition, we present an optional contact-based temporal refinement that refines erroneous per-frame keypoint estimates using contact cues over time. As a result, our HOIL effectively leverages human-object interaction to resolve spatial ambiguity and class imbalance in interaction regions. Codes will be released.

Figures (8)

• Figure 1: Main challenges for 3D human pose estimation from LiDAR points. First, due to spatial ambiguity issue of 3D human and object points, previous SOTA method an2025pre fails to predict accurate pose. Second, class imbalance issue occurs in frequently interacting body parts ($\downarrow$) such as hand and foot, where the number of points are severely low compared to other body parts, object, and background.
• Figure 2: Overall pipeline of HOIL. Given an input point cloud $\mathbf{P}$, we first embed it into input features $\mathbf{F}_\mathrm{p}$ and encode them using a multi-stage encoder with CPPool. The encoded features are then progressively decoded through a multi-stage decoder to produce the final decoder features $\mathbf{F}_{\mathrm{p},\mathrm{dec}}^{(0)}$. At each decoding stage, keypoint queries $\mathbf{Q}$ are iteratively updated via multi-stage keypoint decoder with decoder features. Lastly, HOIL predicts point-level segmentation $\mathbf{S}_\mathrm{p}$ and contact $\mathbf{C}_\mathrm{p}$ from $\mathbf{F}_{\mathrm{p},\mathrm{dec}}^{(0)}$, and 3D keypoint coordinates $\mathbf{K}$ and keypoint-level contact $\mathbf{C}_\mathrm{K}$ from keypoint queries $\mathbf{Q}$.
• Figure 3: Human-object interaction-aware contrastive learning. C and NC refer to contact and non-contact, respectively. Others represent other human body parts except hand and foot. For the simplicity of illustration, we do not divide hand and foot for C and NC whereas our HOICL does operate on those cases as well.
• Figure 4: Qualitative comparison of 3D human pose estimation with the state-of-the-art method an2025pre on Waymo sun2020scalability. Red circles indicate exemplar regions that HOIL outperforms previous methods.
• Figure S1: Correlation between segmentation accuracy and human pose estimation error of state-of-the-art method an2025pre on Waymo sun2020scalability.