OSCAR: Occupancy-based Shape Completion via Acoustic Neural Implicit Representations

TL;DR

An occupancy-based shape completion method that reconstructs complete 3D anatomical geometry from partial ultrasound observations that outperforms state-of-the-art shape completion for B-mode ultrasound by 80% in HD95 score is proposed.

Abstract

Accurate 3D reconstruction of vertebral anatomy from ultrasound is important for guiding minimally invasive spine interventions, but it remains challenging due to acoustic shadowing and view-dependent signal variations. We propose an occupancy-based shape completion method that reconstructs complete 3D anatomical geometry from partial ultrasound observations. Crucially for intra-operative applications, our approach extracts the anatomical surface directly from the image, avoiding the need for anatomical labels during inference. This label-free completion relies on a coupled latent space representing both the image appearance and the underlying anatomical shape. By leveraging a Neural Implicit Representation (NIR) that jointly models both spatial occupancy and acoustic interactions, the method uses acoustic parameters to become implicitly aware of the unseen regions without explicit shadowing labels through tracking acoustic signal transmission. We show that this method outperforms state-of-the-art shape completion for B-mode ultrasound by 80% in HD95 score. We validate our approach both in-silico and on phantom US images with registered mesh models from CT labels, demonstrating accurate reconstruction of occluded anatomy and robust generalization across diverse imaging conditions. Code and data will be released on publication.

Figures (5)

• Figure 1: Proposed framework pipeline.Joint Training: The network couples an acoustic head ($g_{\phi}$) and a geometric head ($s_{\psi}$) via a shared latent space ($\mathcal{Z}$) and backbone ($f_{\theta}$). Supervised by both B-mode synthesis ($\mathcal{L}_{photo}$) and ground-truth shape occupancy ($\mathcal{L}_{occ}$), the ray-based rendering inherently models acoustic shadowing.Test-Time Optimization: At inference, network weights are frozen. A novel latent code $\mathbf{z}^*$ is optimized directly from fragmented B-mode observations without shape labels ($\mathcal{L}_{tto}$). The optimized prior inherently extracts the complete 3D geometry $o(\mathbf{x} \mid \mathbf{z}^*)$.
• Figure 2: Visual analysis of reconstructed meshes. OSCAR shows better structural awareness and accurate completion of invisible anatomy compared to NISF (backbone) and SITD (which strictly requires segmentation inputs).
• Figure 3: Latent interpolation We morph the mean shape into a target shape by linearly interpolating their respective latent codes. The anatomically plausible intermediate states show that our framework learns a continuous and valid geometric prior.
• Figure 4: Bidirectional representation. The acoustic space is accurately predicted by optimizing the latent code $\mathbf{z}$ using only the target shape, demonstrating a strong structural and acoustic coupling.
• Figure 5: Evaluation on phantom data. Our framework accurately completes shapes from real B-modes despite the sim-to-real domain gap. In contrast, SITD avoids this OoD challenge only by requiring explicit point cloud labels.