3D Reconstruction of Shoes for Augmented Reality

TL;DR

This work tackles the limitation of static 2D imagery in online shoe shopping by enabling realistic 3D reconstruction and AR on mobile devices. It leverages 3D Gaussian Splatting, COLMAP-based SfM, and mesh extraction (SuGaR/Sugar) to create view-consistent 3D shoe models from 2D images, supported by a segmentation dataset of 3120 images. The results show a PSNR around the mid-30s dB and IoU around 0.95, with an end-to-end pipeline integrated into a Flutter app and Lens Studio for AR-based virtual try-ons, highlighting practical viability on consumer hardware. The approach offers a scalable path to extend to broader fashion categories and real-time mobile AR shopping.

Abstract

This paper introduces a mobile-based solution that enhances online shoe shopping through 3D modeling and Augmented Reality (AR), leveraging the efficiency of 3D Gaussian Splatting. Addressing the limitations of static 2D images, the framework generates realistic 3D shoe models from 2D images, achieving an average Peak Signal-to-Noise Ratio (PSNR) of 32, and enables immersive AR interactions via smartphones. A custom shoe segmentation dataset of 3120 images was created, with the best-performing segmentation model achieving an Intersection over Union (IoU) score of 0.95. This paper demonstrates the potential of 3D modeling and AR to revolutionize online shopping by offering realistic virtual interactions, with applicability across broader fashion categories.

Figures (9)

• Figure 1: Sample Images from the Dataset
• Figure 3: Extracted Mesh
• Figure 4: Results in AR
• Figure 5: Interactive 3D Viewer
• Figure 6: Overview of Workflow