TransNormal: Dense Visual Semantics for Diffusion-based Transparent Object Normal Estimation

TL;DR

This work tackles monocular surface normal estimation for transparent labware, a challenging problem due to refraction and reflection. It introduces TransNormal, a framework that repurposes Stable Diffusion for single-step normal regression and injects dense visual semantics from DINOv3 via cross-attention to provide robust geometric priors. A new TransNormal-Synthetic dataset supports physics-based rendering of transparent objects and enables controlled evaluation. Across ClearGrasp, TransNormal-Synthetic, and ClearPose, TransNormal achieves state-of-the-art results, notably reducing mean angular error and increasing fine-grained angular accuracy, demonstrating strong zero-shot generalization from synthetic data to real laboratory scenes. This approach offers a promising path for reliable embodied AI in autonomous labs by combining generative priors with semantic guidance and frequency-aware regularization.

Abstract

Monocular normal estimation for transparent objects is critical for laboratory automation, yet it remains challenging due to complex light refraction and reflection. These optical properties often lead to catastrophic failures in conventional depth and normal sensors, hindering the deployment of embodied AI in scientific environments. We propose TransNormal, a novel framework that adapts pre-trained diffusion priors for single-step normal regression. To handle the lack of texture in transparent surfaces, TransNormal integrates dense visual semantics from DINOv3 via a cross-attention mechanism, providing strong geometric cues. Furthermore, we employ a multi-task learning objective and wavelet-based regularization to ensure the preservation of fine-grained structural details. To support this task, we introduce TransNormal-Synthetic, a physics-based dataset with high-fidelity normal maps for transparent labware. Extensive experiments demonstrate that TransNormal significantly outperforms state-of-the-art methods: on the ClearGrasp benchmark, it reduces mean error by 24.4% and improves 11.25° accuracy by 22.8%; on ClearPose, it achieves a 15.2% reduction in mean error. The code and dataset will be made publicly available at https://longxiang-ai.github.io/TransNormal.

Figures (13)

• Figure 1: In-the-wild qualitative results.TransNormal (ours) recovers accurate surface normals on transparent objects. Row 1: Safety goggles with vent holes behind the lens---only our method recovers the flat lens surface. Row 2: Prism under extreme lighting---only ours recovers correct triangular geometry. Row 3: Double-walled glass---ours correctly estimates outer surface normals.
• Figure 2: Overview of the TransNormal framework. (a) Dual-Stream Encoding: the frozen VAE encoder $\mathcal{E}_{\text{vae}}$ extracts RGB latent $\bm{z}_{\text{rgb}}$, while a frozen DINOv3 encoder $\mathcal{E}_{\text{vis}}$ with a trainable linear projector produces semantic conditioning $\bm{c}_{\text{sem}}$; (b) Single-Step Direct Prediction: the fine-tuned SD2 U-Net $f_{{\bm{\theta}}}$ directly regresses the normal latent $\hat{\bm{z}}_{\text{n}}$ from $\bm{z}_{\text{rgb}}$ at a fixed timestep $T$, where the spatial feature $\bm{h}^{(l)}$ at each layer $l$ provides queries $\bm{Q}$, and $\bm{c}_{\text{sem}}$ provides keys $\bm{K}$ and values $\bm{V}$ for cross-attention; (c) Decoding & Wavelet Regularization: the frozen VAE decoder $\mathcal{D}_{\text{vae}}$ reconstructs the predicted normal map $\hat{\bm{N}}$, supervised by latent-space losses ($\mathcal{L}_{\text{rgb}}$, $\mathcal{L}_{\text{normal}}$) and wavelet-domain losses ($\mathcal{L}_{\text{HF}}$, $\mathcal{L}_{\text{LL}}$) that separately penalize high-frequency details and low-frequency structure. (§ \ref{['par:method_overview']})
• Figure 3: Wavelet Edge-Aware Regularization. Haar wavelet decomposes normals into low-frequency (LL) and high-frequency (LH, HL, HH) sub-bands. An edge mask $\bm{M}_{\text{edge}}$ enables: ① LL fidelity for overall shape; ② edge-aligned HF supervision for sharp boundaries.
• Figure 4: Qualitative comparison on transparent object normal estimation. We compare our method against state-of-the-art approaches across TransNormal-Synthetic, ClearGrasp, and ClearPose datasets. For each dataset, the top row shows predicted normals and the bottom row shows angular error maps (blue: low, red: high). Notably, even on ClearPose, an extremely challenging real-world dataset with diverse transparent objects under cluttered scenes, our method achieves superior zero-shot performance compared to other approaches. Existing methods produce blurry or incorrect normals on transparent regions due to refraction, while our method recovers sharp and accurate surface geometry. Please zoom in for details. (§ \ref{['par:qual_comparison']})
• Figure 5: Qualitative ablation study on in-the-wild objects. (a) In-the-wild input RGB image, a transparent cup with a flower inside. (b) Without DINOv3 semantic guidance, the model fails to recognize that the cup is transparent, incorrectly predicting the internal flower as surface geometry. (c) Without wavelet loss, the output exhibits discontinuous artifacts on smooth surfaces. (d) Our full model achieves both correct transparency understanding and smooth, continuous predictions. (§ \ref{['par:ablation_studies']})