HandOcc: NeRF-based Hand Rendering with Occupancy Networks

TL;DR

HandOccness presents a meshless hand rendering framework that replaces parametric meshes with an occupancy-based representation conditioned on a sparse 3D hand skeleton and a NeRF renderer. An occupancy network provides a probabilistic hand surface and features, which combine with appearance embeddings from a CVAE and a deformation model to render novel poses and views. An occupancy-guided sampling strategy and a CNN-based upsampler enable efficient, high-fidelity rendering, including appearance transfer, while handling hand interactions. Evaluations on InterHand2.6M demonstrate state-of-the-art performance, validating the approach's ability to generalize beyond mesh-based initializations and deliver fast, realistic hand rendering in diverse scenarios.

Abstract

We propose HandOcc, a novel framework for hand rendering based upon occupancy. Popular rendering methods such as NeRF are often combined with parametric meshes to provide deformable hand models. However, in doing so, such approaches present a trade-off between the fidelity of the mesh and the complexity and dimensionality of the parametric model. The simplicity of parametric mesh structures is appealing, but the underlying issue is that it binds methods to mesh initialization, making it unable to generalize to objects where a parametric model does not exist. It also means that estimation is tied to mesh resolution and the accuracy of mesh fitting. This paper presents a pipeline for meshless 3D rendering, which we apply to the hands. By providing only a 3D skeleton, the desired appearance is extracted via a convolutional model. We do this by exploiting a NeRF renderer conditioned upon an occupancy-based representation. The approach uses the hand occupancy to resolve hand-to-hand interactions further improving results, allowing fast rendering, and excellent hand appearance transfer. On the benchmark InterHand2.6M dataset, we achieved state-of-the-art results.

Figures (7)

• Figure 1: This figure demonstrates an overview of the proposed approach. Samples are drawn from the occupancy network which is conditioned on the skeletal input. The occupancy model returns per-point probabilities and features of the surface points. The surface points go through a deformation layer that canonicalizes the hand input. Afterwards, per-point occupancy encodings are then given to the NeRF (MLP) along with appearance embeddings. The NeRF renders an RGB image with corresponding features. The final CNN layer upsamples and refines the NeRF output.
• Figure 2: This figure demonstrates the results of the CVAE model. The first column shows input images $\mathbf{I}_{H_i}^A$ and $\mathbf{I}_{H_j}^B$ of two different hands from different people. The second column shows the same hand skeleton $\mathbf{H}_k^X$ rasterized on an RGB image. The output of the model is the last column, showing synthetically generated images $\mathbf{I}_{H_k}^A$, $\mathbf{I}_{H_k}^B$. These images closely resemble the input persons identity in terms of shared features, albeit with different skeleton shape.
• Figure 3: Comparison of single hand rendering to LiveHnad mundra2023livehand and HandAvatar bib:handavatar methods.
• Figure 4: Comparison of interacting hands with Animatable-NeRF peng2021animatable, NeuMan jiang2022neuman, and HandNeRF Guo_2023_CVPR. Each pair of images shows two different views of the same pose. The first row applies NeRF to a novel view, while the second row applies it to a novel pose. The competitor images are sourced from the HandNeRF.
• Figure 5: Figure (a) shows: ground truth, output from the NeRF, output from the upscaling CNN. Figure (b) demonstrates appearance transfer of the same hand pose to different identities.