HumMorph: Generalized Dynamic Human Neural Fields from Few Views

TL;DR

HumMorph advances free-viewpoint rendering of dynamic humans from monocular inputs by removing reliance on precise body shape parameters and by explicitly modeling pose-estimation noise. It conditions a canonical NeRF on a dense, multi-stream feature fusion (global, voxel, and pixel-aligned) via VoluMorph and learns pose-aware deformations through motion weights with forward/backward mappings, refined end-to-end with perceptual and consistency losses. The approach delivers state-of-the-art perceptual quality, especially when using two or more conditioning views, and demonstrates robustness to estimated body parameters, outperforming SHERF and GHuNeRF on HuMMan and DNA-Rendering datasets. Practical impact includes improved, efficient, in-the-wild 3D human synthesis for AR/VR and content creation, with potential extensions to clothing dynamics and camera pose estimation.

Abstract

We introduce HumMorph, a novel generalized approach to free-viewpoint rendering of dynamic human bodies with explicit pose control. HumMorph renders a human actor in any specified pose given a few observed views (starting from just one) in arbitrary poses. Our method enables fast inference as it relies only on feed-forward passes through the model. We first construct a coarse representation of the actor in the canonical T-pose, which combines visual features from individual partial observations and fills missing information using learned prior knowledge. The coarse representation is complemented by fine-grained pixel-aligned features extracted directly from the observed views, which provide high-resolution appearance information. We show that HumMorph is competitive with the state-of-the-art when only a single input view is available, however, we achieve results with significantly better visual quality given just 2 monocular observations. Moreover, previous generalized methods assume access to accurate body shape and pose parameters obtained using synchronized multi-camera setups. In contrast, we consider a more practical scenario where these body parameters are noisily estimated directly from the observed views. Our experimental results demonstrate that our architecture is more robust to errors in the noisy parameters and clearly outperforms the state of the art in this setting.

Figures (10)

• Figure 1: HumMorph is a generalized method for free-viewpoint synthesis of humans in novel poses given a few observations. State-of-the-art methods, including SHERF hu_sherf_2023 and GHuNeRF li_ghunerf_2024, require accurate body pose annotations for the observed views. These are typically unavailable in practice and the poses need to be noisily estimated instead, like in the example above (poses shown in red). In this scenario, existing approaches struggle to model details and synthesize oversmoothed renders. In contrast, our approach includes dense 3D processing modules and accounts for pose estimation errors to accurately recover detail. The letter in parentheses indicates which views were supplied to the methods: L -- left, R -- right or B -- both.
• Figure 2: An overview of our approach. First, we extract the 2D featuremaps $F_t$, which we pass through a VoluMorph module to get the final motion weights $W$. The features $F_t$ and motion weights $W$ are passed to a second VoluMorph module, which outputs the volume $V$ and a global latent code. Finally, we extract $f_\textrm{vox}, f_\textrm{glob}, f_\textrm{pix}$ and combine them using the feature fusion module to condition the NeRF MLP.
• Figure 3: The architecture of our VoluMorph module.
• Figure 4: Qualitative comparison with SHERF (Mo) hu_sherf_2023 and GHuNeRF+ li_ghunerf_2024 given accurate shape and pose parameters. Numbers in parentheses indicate the range of observed views supplied to the respective models.
• Figure 5: Qualitative comparison with SHERF (Mo) hu_sherf_2023 and GHuNeRF+ li_ghunerf_2024 on DNA-Rendering cheng_dna-rendering_2023 with shape and pose parameters estimated from observed views. Numbers in parentheses indicate the range of observed views supplied to the respective models.