KaoLRM: Repurposing Pre-trained Large Reconstruction Models for Parametric 3D Face Reconstruction

TL;DR

KaoLRM tackles the challenge of single-view parametric 3D face reconstruction by re-targeting the pre-trained priors of Large Reconstruction Models (LRMs) to predict FLAME parameters from a single image. It bridges implicit 3D priors with a compact, controllable 3DMM by projecting LRM tri-plane features into FLAME’s parameter space and modeling appearance with surface-bounded 2D Gaussian primitives tightly bound to FLAME geometry. The approach uses an analysis-by-synthesis framework with staged training, incorporating a photometric loss, a geometric binding loss, and regularization terms, achieving improved cross-view consistency and reconstruction accuracy while requiring fewer training samples than prior methods. Experiments on FaceVerse, FFHQ, and NoW demonstrate enhanced accuracy and view robustness, validating the benefit of transferring broad 3D priors from LRM to a practical parametric head model for real-world, multi-view scenarios.

Abstract

We propose KaoLRM to re-target the learned prior of the Large Reconstruction Model (LRM) for parametric 3D face reconstruction from single-view images. Parametric 3D Morphable Models (3DMMs) have been widely used for facial reconstruction due to their compact and interpretable parameterization, yet existing 3DMM regressors often exhibit poor consistency across varying viewpoints. To address this, we harness the pre-trained 3D prior of LRM and incorporate FLAME-based 2D Gaussian Splatting into LRM's rendering pipeline. Specifically, KaoLRM projects LRM's pre-trained triplane features into the FLAME parameter space to recover geometry, and models appearance via 2D Gaussian primitives that are tightly coupled to the FLAME mesh. The rich prior enables the FLAME regressor to be aware of the 3D structure, leading to accurate and robust reconstructions under self-occlusions and diverse viewpoints. Experiments on both controlled and in-the-wild benchmarks demonstrate that KaoLRM achieves superior reconstruction accuracy and cross-view consistency, while existing methods remain sensitive to viewpoint variations. The code is released at https://github.com/CyberAgentAILab/KaoLRM.

Figures (6)

• Figure 1: KaoLRM predicts FLAME parameters from a single-view facial image and yields more faithful and consistent predictions for the same subject captured from different viewpoints. We validate the consistency qualitatively through visual inspection of the reconstructed FLAME meshes and quantitatively by measuring the variance of the FLAME parameters across the views. The methods for reference are DECA feng2021learning and SMIRK retsinas20243d.
• Figure 2: Motivation of KaoLRM. Instead of learning 3DMM regressors from scratch using multi-view face datasets that are expensive to scale, we leverage the learned 3D prior of LRM hong2024lrm (e.g., tri-plane features), allowing KaoLRM to be trained with a moderate amount of data.
• Figure 3: Overall architecture of KaoLRM. Top: KaoLRM leverages the pre-trained image-to-triplane transformer of LRM hong2024lrm, and trains an additional network to predict FLAME parameters from the triplane features. The resulting FLAME mesh is then converted to 2D Gaussian primitives for synthesizing input-view or novel-view images. Bottom: After obtaining the FLAME mesh, we sample points via barycentric interpolation to initialize the centers of 2D Gaussians. At each Gaussian center, the corresponding triplane feature is queried and decoded into Gaussian attributes.
• Figure 4: Per-vertex variances in local geometries. The heatmap is plotted on the mean face of multi-view predictions after keypoint alignment. Ours achieves more consistent predictions across viewpoints.
• Figure 5: Qualitative comparison on the test set of FFHQ dataset karras2019style. We align the results of different methods via an estimated transformation of the keypoints. Ours reconstructs shapes more accurately and recovers expressions more faithfully.