ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image

TL;DR

This paper introduces ANIM, a novel method that reconstructs arbitrary 3D human shapes from single-view RGB-D images with an unprecedented level of accuracy and enhances the quality of the reconstructed shape by introducing a depth-supervision strategy, which improves the accuracy of the signed distance field estimation of points that lie on the re-constructed surface.

Abstract

Recent progress in human shape learning, shows that neural implicit models are effective in generating 3D human surfaces from limited number of views, and even from a single RGB image. However, existing monocular approaches still struggle to recover fine geometric details such as face, hands or cloth wrinkles. They are also easily prone to depth ambiguities that result in distorted geometries along the camera optical axis. In this paper, we explore the benefits of incorporating depth observations in the reconstruction process by introducing ANIM, a novel method that reconstructs arbitrary 3D human shapes from single-view RGB-D images with an unprecedented level of accuracy. Our model learns geometric details from both multi-resolution pixel-aligned and voxel-aligned features to leverage depth information and enable spatial relationships, mitigating depth ambiguities. We further enhance the quality of the reconstructed shape by introducing a depth-supervision strategy, which improves the accuracy of the signed distance field estimation of points that lie on the reconstructed surface. Experiments demonstrate that ANIM outperforms state-of-the-art works that use RGB, surface normals, point cloud or RGB-D data as input. In addition, we introduce ANIM-Real, a new multi-modal dataset comprising high-quality scans paired with consumer-grade RGB-D camera, and our protocol to fine-tune ANIM, enabling high-quality reconstruction from real-world human capture.

Figures (16)

• Figure 1: ANIM enables human shape reconstruction with higher accuracy and without shape distortions compared to the state-of-the-art methods based on monocular RGB-D or RGB input.
• Figure 2: ANIM architecture. Our proposed framework has three major components: i) a multi-resolution appearance feature extractor for color and normal inputs (LR-FE and HR-FE), ii) a novel SparseConvNet U-Net (Volume Feature Extractor or VFE) that efficiently extracts geometry features from 3D voxels and low-resolution image features, iii) an MLP that estimate the implicit surface representation of full-body humans. $+$ denotes concatenation, $\Pi$ means fetching pixel-aligned 2D LR features and concatenating with 3D voxels, and $\nabla$ indicates gradient operation applied to retrieve normals from depth map (using neighboring pixel cross-product).
• Figure 3: ANIM reconstructions from real-world capture with a consumer-grade RGB-D camera (Azure Kinect) before (a), and after (b) fine-tuning on the proposed dataset ANIM-Real, which quality is closer to a high-res scan capture (c).
• Figure 4: Semantic-aware sampling. Compared to uniform sampling (left), semantic-aware sampling (right) enables finer learning of human features on specific regions such as the head and hands.
• Figure 5: ANIM reconstructs fine-level cloth details such as wrinkles on the cloth and body with high accuracy even when the input is a consumer-grade RGB-D camera (Azure Kinect).