Learning Neural Parametric 3D Breast Shape Models for Metrical Surface Reconstruction From Monocular RGB Videos

TL;DR

This work addresses the need for accurate, accessible 3D breast surface modeling by introducing liRBSM, a localized neural implicit representation that decomposes the breast surface into multiple region-specific SDFs anchored at anatomical landmarks. It couples this model with a low-cost monocular RGB video reconstruction pipeline that uses an off-the-shelf SfM (VGGSfM) to produce metrically correct surfaces within minutes, without specialized hardware. The key contributions are: (i) a six-anchor local SDF ensemble with background conditioning, (ii) an auto-decoder training regime on 3D breast scans, and (iii) a full monocular-video surface reconstruction workflow with landmark-based alignment and model fitting, all openly available. The results show substantial improvements over prior global implicit methods (iRBSM) and PCA-based models (RBSM), achieving less than $2$ mm surface error and detailed features like skin folds and nipples, enabling practical clinical and consumer applications with transparency and reproducibility.

Abstract

We present a neural parametric 3D breast shape model and, based on this model, introduce a low-cost and accessible 3D surface reconstruction pipeline capable of recovering accurate breast geometry from a monocular RGB video. In contrast to widely used, commercially available yet prohibitively expensive 3D breast scanning solutions and existing low-cost alternatives, our method requires neither specialized hardware nor proprietary software and can be used with any device that is able to record RGB videos. The key building blocks of our pipeline are a state-of-the-art, off-the-shelf Structure-from-motion pipeline, paired with a parametric breast model for robust and metrically correct surface reconstruction. Our model, similarly to the recently proposed implicit Regensburg Breast Shape Model (iRBSM), leverages implicit neural representations to model breast shapes. However, unlike the iRBSM, which employs a single global neural signed distance function (SDF), our approach -- inspired by recent state-of-the-art face models -- decomposes the implicit breast domain into multiple smaller regions, each represented by a local neural SDF anchored at anatomical landmark positions. When incorporated into our surface reconstruction pipeline, the proposed model, dubbed liRBSM (short for localized iRBSM), significantly outperforms the iRBSM in terms of reconstruction quality, yielding more detailed surface reconstruction than its global counterpart. Overall, we find that the introduced pipeline is able to recover high-quality 3D breast geometry within an error margin of less than 2 mm. Our method is fast (requires less than six minutes), fully transparent and open-source, and -- together with the model -- publicly available at https://rbsm.re-mic.de/local-implicit.

Figures (12)

• Figure 1: Contributions. We propose a high-fidelity 3D parametric breast model along with a model-based surface reconstruction pipeline that is able to accurately reconstruct 3D breast surfaces from a single monocular RGB video. Our model, trained on over 160 breast scans, builds upon a localized neural implicit representation and represents breast geometry as an ensemble of local MLPs instead of a single global network, significantly improving the level of detail.
• Figure 2: Overview of our model's architecture. Instead of using a single global MLP to represent breast shapes, we partition the implicit domain into six smaller regions, centered around anchor points $\{a_1,a_2,\dots,a_6\}$ placed at anatomical landmark positions. Each region is represented by a shallow MLP $\phi_i$, conditioned on a local latent code $z_i$. To account for areas distant from any anchor, we introduce an additional MLP $\phi_0$ that shares the same representation as the local MLPs. The final SDF $\phi(x,z)$ at a query point $x$ is obtained by blending the outputs of all MLPs using a Gaussian weighting scheme. An illustration of the spatial influence of each local MLP is shown on the left, visualized on the resulting model's mean shape.
• Figure 3: Surface reconstruction from monocular RGB videos. For the first time, we enable low-cost and accurate breast surface reconstruction from just a single monocular RGB video, acquired using commodity hardware (such as smartphones). Our pipeline starts by applying Structure-from-motion to a set of frames extracted from the input video, resulting in a sparse point cloud and camera parameters. We then align the estimated point cloud to our model's mean shape, prune away background points, and finally fit our real-world-scale model to the resulting point cloud. Depending on the available hardware, the entire reconstruction can be executed in under six minutes.
• Figure 4: Latent space interpolation. We linearly interpolate between the latent codes corresponding to the shapes outlined in blue. Starting from the mean shape at the center, we interpolate in positive direction along the first and second principal components on the right, and in the negative direction on the left. Principal components are derived by applying PCA to the optimized latent codes from the training data. Our latent space is continous and well-behaved.
• Figure 5: Random samples from our model. The proposed model is highly expressive and allows for a variety of plausible breast shapes.