Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces

TL;DR

This paper tackles the challenge of reconstructing geometry and enabling novel view synthesis for challenging, non-Lambertian surfaces under limited viewpoints. It introduces Tactile-Informed 3D Gaussian Splatting (3DGS), which fuses RGB multi-view data with local depth maps from tactile sensing to regularize 3D transmittance around touch locations and employs an edge-aware, proximity-masked smoothness term. The approach achieves state-of-the-art geometry on glossy and reflective objects, with notably faster reconstruction (roughly 10x) and improved performance in minimal-view scenarios compared to NeRF-based and standalone 3DGS baselines. Evaluations on Shiny Blender, Glossy Synthetic, and real-world tactile-vision data demonstrate robust surface reconstruction and competitive novel view synthesis, highlighting the practical benefits for robotics and 3D modeling in challenging material conditions.

Abstract

Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.

Figures (7)

• Figure 1: We combine multi-view visual data and tactile sensing information within a 3D Gaussian Splatting framework for accurate geometry reconstruction and novel view synthesis of challenging surfaces. On the left, a robotic arm is equipped with tactile sensors. On the right, we show results on novel view synthesis and depth reconstruction, using a five minimal-view setting and tactile input (first row) -- touches not only improve surface reconstruction, but also strengthen novel view synthesis.
• Figure 2: Proximity-Based Mask's impact on the gradients computed by the edge-aware smoothness loss. Left: points collected on the object's surface alongside the derived proximity mask. Centre: averaged horizontal and vertical gradients as determined by the smoothness loss, where lighter shades correspond to higher gradients. Right: integration of the smoothness loss with the proximity mask using two distinct approaches: (a) implementation of a distance-based Gaussian decay within the proximity mask, and (b) masking based on a discrete threshold from the contact surface.
• Figure 3: Surface reconstruction qualitative results from 5 training views on the Glossy Synthetic dataset
• Figure 4: Novel-view synthesis qualitative results from 5 training views on the Glossy Synthetic dataset
• Figure 5: Rasterised depth maps of the Toaster object. Our method results in a smoother, accurate reconstruction compared to 3DGS and 3DGS with additional regularisation on the smoothness loss.