VersaQ-3D: A Reconfigurable Accelerator Enabling Feed-Forward and Generalizable 3D Reconstruction via Versatile Quantization

TL;DR

VersaQ-3D tackles the bottleneck of on-device 3D reconstruction with VGGT by delivering calibration-free, versatile 4-bit quantization and a reconfigurable accelerator. The approach combines offline orthogonal transforms (WHT and DCT) to decorrelate activations and preserve weight structure, enabling accurate INT4/INT8/BF16 inference without calibration data. A unified, multi-precision datapath and a two-stage recomputation tiling address long-sequence global attention, reducing memory and latency while preserving accuracy (98–99% of full precision at W4A8; 1.61×–2.39× gains at W4A4 over prior PTQ methods). Hardware results show 5.2×–10.8× speedups and substantial energy efficiency improvements over edge GPUs, demonstrating the practicality of instant 3D reconstruction for AR/VR, robotics, and autonomous systems.

Abstract

The Visual Geometry Grounded Transformer (VGGT) enables strong feed-forward 3D reconstruction without per-scene optimization. However, its billion-parameter scale creates high memory and compute demands, hindering on-device deployment. Existing LLM quantization methods fail on VGGT due to saturated activation channels and diverse 3D semantics, which cause unreliable calibration. Furthermore, VGGT presents hardware challenges regarding precision-sensitive nonlinear operators and memory-intensive global attention. To address this, we propose VersaQ-3D, an algorithm-architecture co-design framework. Algorithmically, we introduce the first calibration-free, scene-agnostic quantization for VGGT down to 4-bit, leveraging orthogonal transforms to decorrelate features and suppress outliers. Architecturally, we design a reconfigurable accelerator supporting BF16, INT8, and INT4. A unified systolic datapath handles both linear and nonlinear operators, reducing latency by 60%, while two-stage recomputation-based tiling alleviates memory pressure for long-sequence attention. Evaluations show VersaQ-3D preserves 98-99% accuracy at W4A8. At W4A4, it outperforms prior methods by 1.61x-2.39x across diverse scenes. The accelerator delivers 5.2x-10.8x speedup over edge GPUs with low power, enabling efficient instant 3D reconstruction.

Figures (14)

• Figure 1: Llama and VGGT activation value distributions and corresponding percentile distributions. The red band (25th-75th percentiles) denotes the inter-quartile range containing the middle 50% of activations. Unlike LLMs, which show isolated spiking outliers, VGGT exhibits saturated activation channels where many channels maintain persistently high activation values.
• Figure 2: VGGT model structure. A DINO-based feature extractor feeds an AA module that interleaves frame and global attention over tokens reshaped between $S\times[P,C]$ and $[S\times P, C]$ for intra-frame feature normalization and inter-frame information fusion.
• Figure 3: Inference runtime breakdown on (a) different GPUs with S=3 and (b) different sequence lengths S on Jetson Orin NX, evaluated on the Example/Kitchen dataset.
• Figure 4: Salient distribution in VGGT. Here we take channel variance to indicate the saliency of the activation. The variance remains considerable after Hadamard rotation.
• Figure 5: VersaQ‑3D quantization framework. The attention module (with MLP blocks treated analogously) follows a four‑stage pipeline that combines offline weight preparation with online activation processing for low‑precision inference. Stage 1: Fuse LayerNorm into weights and activations, and apply $Q$, $K$, and $V$ projections whose weights are pre‑processed offline using Hadamard transforms, scaling, and DCT. Stage 2: Apply IDCT, dequantize to BF16 for RoPE, then apply WHT and re‑quantize to INT. Stage 3: Perform MHA in INT using quantized $Q$, $K$, and $V$. Stage 4: Apply the same offline preparation to the output projection weights, followed by IDCT and LayerScale fusion, while keeping activations in the rotated domain for the next layer to reduce online WHT operations and on‑chip computation.