VIAFormer: Voxel-Image Alignment Transformer for High-Fidelity Voxel Refinement

TL;DR

VIAFormer introduces a multi-view conditioned voxel refinement framework that tightly integrates 3D voxel priors with 2D image guidance. It combines Correctional Flow to refine latent representations along a direct trajectory, an Image Index for explicit 3D grounding of image tokens, and a Hybrid Stream Transformer to fuse modalities within a Voxel-Image Union-Space. Across VFM-derived and synthetic corruptions, VIAFormer achieves state-of-the-art results on mid-resolution $64^3$ voxels and demonstrates practical utility as a bridge in real-world 3D creation pipelines. This work advances voxel-based methods toward the large-model, big-data era by enabling robust, geometry-aware, multi-view refinement with strong generalization guarantees.

Abstract

We propose VIAFormer, a Voxel-Image Alignment Transformer model designed for Multi-view Conditioned Voxel Refinement--the task of repairing incomplete noisy voxels using calibrated multi-view images as guidance. Its effectiveness stems from a synergistic design: an Image Index that provides explicit 3D spatial grounding for 2D image tokens, a Correctional Flow objective that learns a direct voxel-refinement trajectory, and a Hybrid Stream Transformer that enables robust cross-modal fusion. Experiments show that VIAFormer establishes a new state of the art in correcting both severe synthetic corruptions and realistic artifacts on the voxel shape obtained from powerful Vision Foundation Models. Beyond benchmarking, we demonstrate VIAFormer as a practical and reliable bridge in real-world 3D creation pipelines, paving the way for voxel-based methods to thrive in large-model, big-data wave.

Figures (16)

• Figure 1: The Conditioned Voxel Refinement Task. VIAFormer learns to refine a wide range of imperfections guided by multi-view images (Cond.). It handles both (a) various synthetic corruptions and (b) realistic degradations from Vision Foundation Models' outputs, producing clean and complete voxel grids.
• Figure 2: Overview of VIAFormer Architecture. An incomplete noisy voxel grid (derived from VFMs or synthetic corruptions) and multi-view images are encoded into latent tokens. Our Image Index provides explicit 3D spatial locations to the 2D image patch tokens, creating a Union-Space for effective cross-modal fusion within a Hybrid Stream Transformer. The transformer, trained with a Correctional Flow objective, predicts a refinement velocity vector. This vector guides an ODE solver flowmatchingcode24 to produce a clean latent, which the VAE decoder converts into the final refined voxel grid.
• Figure 3: Attention Map Visualization. (Left) Standard cross-attention exhibits Attention Collapse, failing to learn spatial correspondence. (Middle and Right) The Dual Attention and Single Attention of our model, respectively, operate within the Voxel-Image Union-Space to enable structured, bidirectional attention.
• Figure 4: Illustration of the Image Index and Union-Space Correspondence. The Image Index is generated by rendering the input voxel grid $\tilde{v}$ with its 3D coordinates encoded as colors, thereby assigning a spatial location to each 2D image patch token (top). This explicit geometric grounding enables spatially-aware cross-attention between voxel and image tokens, allowing the model to perform effective multi-view guided refinement (bottom).
• Figure 5: Visual Comparison of Voxel Refinement on VFM-Derived Inputs. The error map highlights true positives (green), false positives (red), and false negatives (blue). Compared to the input (Pi3 pi3_25) and other methods, VIAFormer consistently generates more complete and cleaner geometry with substantially fewer artifacts.