Robust Human Registration with Body Part Segmentation on Noisy Point Clouds

TL;DR

SegFit addresses robustly registering parametric human models to noisy 3D point clouds by unifying body-part segmentation with SMPL-X fitting. It employs an initial part-based segmentation to guide centroid-based initialization, followed by a two-stage SMPL-X optimization under a VPoser pose prior and a multi-term objective that balances data fidelity, pose realism, and shape regularization, then refines segmentation via model-based nearest-neighbor voting. Evaluations on InterCap, EgoBody, and BEHAVE show SegFit achieves state-of-the-art pose and segmentation accuracy, with substantial improvements and faster runtimes than prior methods, and enables self-supervised fine-tuning of segmentation networks. This hybrid approach improves robustness in cluttered, occluded real-world scenes, enabling more faithful, real-time 3D human representations for AR, HRI, and human-object interaction tasks.

Abstract

Registering human meshes to 3D point clouds is essential for applications such as augmented reality and human-robot interaction but often yields imprecise results due to noise and background clutter in real-world data. We introduce a hybrid approach that incorporates body-part segmentation into the mesh fitting process, enhancing both human pose estimation and segmentation accuracy. Our method first assigns body part labels to individual points, which then guide a two-step SMPL-X fitting: initial pose and orientation estimation using body part centroids, followed by global refinement of the point cloud alignment. Additionally, we demonstrate that the fitted human mesh can refine body part labels, leading to improved segmentation. Evaluations on the cluttered and noisy real-world datasets InterCap, EgoBody, and BEHAVE show that our approach significantly outperforms prior methods in both pose estimation and segmentation accuracy. Code and results are available on our project website: https://segfit.github.io

Figures (5)

• Figure 1: Our method SegFit reconstructs human poses from point clouds using body part segmentation and the SMPL-X model pavlakos2019expressive. We showcase SMPL-X fitting results on the EgoBody dataset zhang2022egobody, and compare to the state-of-the-art methods ArtEq arteq and NICP marin2025nicp.
• Figure 2: The 15 body parts and their centroids (left). Example body parts segmentation of two humans from our SegFit (right).
• Figure 3: Datasets. Example scenes from BEHAVE bhatnagar22behave, EgoBody zhang2022egobody, and InterCap huang2024intercap. We show RGB images for illustration only, all experiments are performed on single-view depth maps.
• Figure 4: Refined Body-Part Segmentation. Example output of our SegFit on the InterCap huang2024intercap dataset. We show the initial body-part segmentation from Human3D Takmaz_2023_ICCV(left), our registered human mesh (center), and the improvied human body-part segmentation based on nearest neighbors majority voting (right). Notice that, at first, the suitecase is mistakenly labeled as the left leg. However, despite the occlusion, our approach successfully corrects the human pose and refines the body-part segmentation.
• Figure 5: Qualitative Results. Example outputs of our SegFit on the EgoBody zhang2022egobody dataset. From left to right: the input single-view point cloud showing the full scene including multiple humans, clutter and background, registered human meshes by SegFit from the front and side perspective, the refined body-part segmentation by SegFit. See Section \ref{['sec:experiments_qualitative']} for additional details.