PocketGS: On-Device Training of 3D Gaussian Splatting for High Perceptual Modeling

TL;DR

PocketGS tackles the challenge of training 3D Gaussian Splatting on commodity mobile devices under stringent budgets by introducing three co-designed operators: geometry-prior construction (G), prior-conditioned Gaussian parameterization (I), and hardware-aligned differentiable splatting (T). These components address three core contradictions—input unreliability, initialization-convergence, and hardware differentiability—enabling stable end-to-end on-device training. The method leverages a GPU-native global BA, a single-reference MVS, and disc-like anisotropic Gaussian seeding to produce a compact, geometry-faithful prior and well-conditioned Gaussians, all executed entirely on GPU on-device. Experimental results across LLFF, NeRF-Synthetic, and MobileScan demonstrate competitive or superior perceptual quality with significantly lower end-to-end training time and memory footprint, making a practical capture-to-render pipeline feasible on smartphones.

Abstract

Efficient and high-fidelity 3D scene modeling is a long-standing pursuit in computer graphics. While recent 3D Gaussian Splatting (3DGS) methods achieve impressive real-time modeling performance, they rely on resource-unconstrained training assumptions that fail on mobile devices, which are limited by minute-scale training budgets and hardware-available peak-memory. We present PocketGS, a mobile scene modeling paradigm that enables on-device 3DGS training under these tightly coupled constraints while preserving high perceptual fidelity. Our method resolves the fundamental contradictions of standard 3DGS through three co-designed operators: G builds geometry-faithful point-cloud priors; I injects local surface statistics to seed anisotropic Gaussians, thereby reducing early conditioning gaps; and T unrolls alpha compositing with cached intermediates and index-mapped gradient scattering for stable mobile backpropagation. Collectively, these operators satisfy the competing requirements of training efficiency, memory compactness, and modeling fidelity. Extensive experiments demonstrate that PocketGS is able to outperform the powerful mainstream workstation 3DGS baseline to deliver high-quality reconstructions, enabling a fully on-device, practical capture-to-rendering workflow.

Figures (9)

• Figure 1: Our PocketGS enables high-quality end-to-end 3DGS reconstruction on commodity smartphones. Compared to standard 3DGS workstation baselines, PocketGS achieves superior visual fidelity (LPIPS: 0.108) within a tight training budget ( 500 iterations, $\sim$4 minutes on an iPhone 15).
• Figure 2: Overview of the PocketGS framework. PocketGS tackles on-device 3DGS training through three coupled operators: ($\mathcal{G}$) Geometry Prior Construction employs an information-gated gate for frame selection, followed by GPU-native Schur-complement BA and single-reference MVS. The MVS module constructs a 3D cost volume by sampling depth hypotheses ${d_1, \dots, d_n}$ via census matching to produce a dense geometric scaffold. ($\mathcal{I}$) Prior-Conditioned Parameterization seeds anisotropic Gaussians by estimating local surface statistics (normals $\mathbf{s}_n$ and scales) to front-load structure discovery via disc-like covariance seeding. ($\mathcal{T}$) Hardware-Aligned Splatting implements a mobile-native differentiable renderer using unrolled alpha-compositing ($S = {C_{in}, \alpha}$) and index-mapped gradient scattering to ensure stable backpropagation within tight mobile memory bounds of the canonical parameter buffer ($\mu, \Sigma, c, \alpha$).
• Figure 3: Qualitative comparison on our MobileScan dataset. PocketGS (Ours) consistently recovers sharper textures and finer details compared to baselines. Notably, 3DGS-MVS-WK suffers from artifacts despite its high-density prior, as the over-concentrated initial points impede effective Gaussian redistribution within a limited iteration budget. Our method avoids this bottleneck through prior-conditioned initialization, achieving superior structural fidelity (e.g., bicycle spokes) that closely matches the ground truth (GT).
• Figure 4: Qualitative comparison on the LLFF dataset. Our method produces much sharper textures and more accurate thin structures (e.g., the leaves in Fern and the petals in Flower), closely matching the ground truth (GT).
• Figure 5: Qualitative comparison on the NeRF Synthetic dataset. In these object-centric synthetic scenes, our method produces high-fidelity results.