Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mask-Conditioned Voxel Diffusion for Joint Geometry and Color Inpainting

Aarya Sumuk

TL;DR

This work tackles the restoration of damaged 3D objects by jointly reconstructing geometry and color within a voxel framework. It introduces a two-stage approach: first predicting a damage mask from 2D RGB slices, then performing mask-conditioned diffusion inpainting on $32^3$ voxels to recover occupancy and color while preserving observed regions. The method outperforms symmetry-based baselines in geometry completion and yields coherent color under synthetic damage on textured artifacts, demonstrating the practicality of explicit mask conditioning for volumetric diffusion in restoration tasks. While effective at the chosen resolution, color fidelity remains sensitive to texture complexity, motivating future work on higher-resolution representations and stronger appearance priors for real-world cultural heritage applications.

Abstract

We present a lightweight two-stage framework for joint geometry and color inpainting of damaged 3D objects, motivated by the digital restoration of cultural heritage artifacts. The pipeline separates damage localization from reconstruction. In the first stage, a 2D convolutional network predicts damage masks on RGB slices extracted from a voxelized object, and these predictions are aggregated into a volumetric mask. In the second stage, a diffusion-based 3D U-Net performs mask-conditioned inpainting directly on voxel grids, reconstructing geometry and color while preserving observed regions. The model jointly predicts occupancy and color using a composite objective that combines occupancy reconstruction with masked color reconstruction and perceptual regularization. We evaluate the approach on a curated set of textured artifacts with synthetically generated damage using standard geometric and color metrics. Compared to symmetry-based baselines, our method produces more complete geometry and more coherent color reconstructions at a fixed 32^3 resolution. Overall, the results indicate that explicit mask conditioning is a practical way to guide volumetric diffusion models for joint 3D geometry and color inpainting.

Mask-Conditioned Voxel Diffusion for Joint Geometry and Color Inpainting

TL;DR

This work tackles the restoration of damaged 3D objects by jointly reconstructing geometry and color within a voxel framework. It introduces a two-stage approach: first predicting a damage mask from 2D RGB slices, then performing mask-conditioned diffusion inpainting on voxels to recover occupancy and color while preserving observed regions. The method outperforms symmetry-based baselines in geometry completion and yields coherent color under synthetic damage on textured artifacts, demonstrating the practicality of explicit mask conditioning for volumetric diffusion in restoration tasks. While effective at the chosen resolution, color fidelity remains sensitive to texture complexity, motivating future work on higher-resolution representations and stronger appearance priors for real-world cultural heritage applications.

Abstract

We present a lightweight two-stage framework for joint geometry and color inpainting of damaged 3D objects, motivated by the digital restoration of cultural heritage artifacts. The pipeline separates damage localization from reconstruction. In the first stage, a 2D convolutional network predicts damage masks on RGB slices extracted from a voxelized object, and these predictions are aggregated into a volumetric mask. In the second stage, a diffusion-based 3D U-Net performs mask-conditioned inpainting directly on voxel grids, reconstructing geometry and color while preserving observed regions. The model jointly predicts occupancy and color using a composite objective that combines occupancy reconstruction with masked color reconstruction and perceptual regularization. We evaluate the approach on a curated set of textured artifacts with synthetically generated damage using standard geometric and color metrics. Compared to symmetry-based baselines, our method produces more complete geometry and more coherent color reconstructions at a fixed 32^3 resolution. Overall, the results indicate that explicit mask conditioning is a practical way to guide volumetric diffusion models for joint 3D geometry and color inpainting.
Paper Structure (36 sections, 12 equations, 7 figures, 2 tables)

This paper contains 36 sections, 12 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Geometric Reconstruction and Completion
  4. Slice-Based and 2.5D Processing
  5. Diffusion Models for 3D Shape Completion
  6. Color and Appearance Inpainting
  7. Methods
  8. Voxel Representation
  9. Stage 1: Damage Mask Prediction
  10. Architecture and training.
  11. Stage 2: Mask-Conditioned Diffusion Inpainting
  12. Network architecture.
  13. Training objective.
  14. Optimization.
  15. Dataset and Preprocessing
  16. ...and 21 more sections

Figures (7)

  • Figure 1: Representative meshes from the curated artifact dataset.
  • Figure 2: Example mesh and corresponding $32^3$ voxel occupancy grid.
  • Figure 3: Example axial voxel slices showing intact geometry, synthetic damage mask, and damaged input.
  • Figure 4: Stage 1 mask prediction: (left) RGB slice, (center) ground-truth mask overlay, (right) predicted mask overlay.
  • Figure 5: 3D inpainting for an artifact with a distinct pattern: damaged input versus diffusion reconstruction and ground truth (as rendered in the figure).
  • ...and 2 more figures