HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields
Arnab Dey, Di Yang, Antitza Dantcheva, Jean Martinet
TL;DR
HFNeRF tackles the lack of explicit human skeleton structure in NeRF-based avatars by introducing a generalizable framework that learns biomechanic features from 2D image encoders and injects them into 3D NeRF representations. The method predicts joint heatmaps through an auxiliary head that processes intermediate NeRF features, enabling differentiable learning of color, geometry, and skeleton from sparse views. Evaluations on the RenderPeople dataset show encoder choice impacts, with ResNet providing better render quality and DINO yielding more accurate heatmaps via OpenPose-based distillation. The approach offers a pathway to skeleton-aware virtual avatars for AR/VR and can extend to other biomechanic properties beyond skeleton detection.
Abstract
In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.