Link to the Past: Temporal Propagation for Fast 3D Human Reconstruction from Monocular Video

TL;DR

TemPoFast3D tackles real-time 3D clothed human reconstruction from monocular video by exploiting temporal coherence through a canonical-space framework tied to SMPL-based coordinate mapping. It blends pixel-aligned implicit functions with a propagating canonical representation, using strategies like volumetric boundary filtering, visibility-guided sampling, and surface-adjacent sampling to drastically reduce per-frame computation while preserving geometry and texture quality. The approach is plug-and-play with existing backbones and scales to multi-view input, achieving up to 12 FPS on a single GPU and competitive quantitative metrics against state-of-the-art methods. This work enables practical, temporally coherent 3D reconstructions for applications in VR, telepresence, and HCI without subject-specific templates or extensive per-video optimization.

Abstract

Fast 3D clothed human reconstruction from monocular video remains a significant challenge in computer vision, particularly in balancing computational efficiency with reconstruction quality. Current approaches are either focused on static image reconstruction but too computationally intensive, or achieve high quality through per-video optimization that requires minutes to hours of processing, making them unsuitable for real-time applications. To this end, we present TemPoFast3D, a novel method that leverages temporal coherency of human appearance to reduce redundant computation while maintaining reconstruction quality. Our approach is a "plug-and play" solution that uniquely transforms pixel-aligned reconstruction networks to handle continuous video streams by maintaining and refining a canonical appearance representation through efficient coordinate mapping. Extensive experiments demonstrate that TemPoFast3D matches or exceeds state-of-the-art methods across standard metrics while providing high-quality textured reconstruction across diverse pose and appearance, with a maximum speed of 12 FPS.

Figures (12)

• Figure 1: We propose TemPoFast3D, a novel pipeline to leverage the temporal coherency of human appearance for efficient and accurate 3D human reconstruction from monocular videos. We temporally propagate information from the past frames result by blending the pixel-aligned implicit function and avatar reconstruction method.
• Figure 2: Overview of our TemPoFast3D pipeline. Given an input RGB frame $I_t$, our method combines efficient canonical space processing with coordinate mapping for fast 3D human reconstruction. The pipeline consists of: (Section \ref{['ssec:preliminary']}) Feature extraction and SMPL params regression, (Section \ref{['ssec:canspace_inference']}) Mapping canonical coordinates to posed space, (Section \ref{['pg:posed']}) Shape and color query, and (Section \ref{['ssec:temp_pro']}) Canonical space processing. The canonical space representation $X_c$ is continuously updated across frames.
• Figure 3: Warped sampling coordinate visualization. Sampling points (gray) transition from uniform distribution in canonical space (left) to non-uniform distribution after deformation (right), demonstrating how our coordinate mapping affects sampling density around the SMPL mesh (blue).
• Figure 4: Effect of volumetric boundary filtering. (a) Reconstructed meshes without filtering in canonical (top) and deformed pose (bottom) show artifacts. (b) Volumetric boundary mask. (c) Filtered reconstruction results show cleaner geometry in both poses, eliminating artifacts beyond the valid body region.
• Figure 5: Qualitative comparison of geometry reconstruction quality. The top two rows show results on the CAPE dataset CAPEdataset, while the bottom two rows are from the THuman2.0 dataset thuman2dataset. For best viewing, please zoom in on a digital screen.