UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations

TL;DR

This paper addresses driving scene reconstruction using multiple camera models by introducing UniGaussian, which learns a unified 3D Gaussian representation from pinhole and fisheye inputs. It contributes a differentiable rendering method that distorts 3D Gaussians via affine transformations to match fisheye distortions, and a framework that jointly optimizes a shared Gaussian scene across sensors and modalities (depth, semantic, normals, and LiDAR). The approach yields holistic scene understanding with real-time rendering and demonstrates superior rendering quality and speed across fisheye and pinhole inputs, including multi-camera scenarios. This work enables more accurate and scalable simulation for autonomous driving, bridging gaps between camera models and modalities in 3D Gaussian splatting-based reconstruction.

Abstract

Urban scene reconstruction is crucial for real-world autonomous driving simulators. Although existing methods have achieved photorealistic reconstruction, they mostly focus on pinhole cameras and neglect fisheye cameras. In fact, how to effectively simulate fisheye cameras in driving scene remains an unsolved problem. In this work, we propose UniGaussian, a novel approach that learns a unified 3D Gaussian representation from multiple camera models for urban scene reconstruction in autonomous driving. Our contributions are two-fold. First, we propose a new differentiable rendering method that distorts 3D Gaussians using a series of affine transformations tailored to fisheye camera models. This addresses the compatibility issue of 3D Gaussian splatting with fisheye cameras, which is hindered by light ray distortion caused by lenses or mirrors. Besides, our method maintains real-time rendering while ensuring differentiability. Second, built on the differentiable rendering method, we design a new framework that learns a unified Gaussian representation from multiple camera models. By applying affine transformations to adapt different camera models and regularizing the shared Gaussians with supervision from different modalities, our framework learns a unified 3D Gaussian representation with input data from multiple sources and achieves holistic driving scene understanding. As a result, our approach models multiple sensors (pinhole and fisheye cameras) and modalities (depth, semantic, normal and LiDAR point clouds). Our experiments show that our method achieves superior rendering quality and fast rendering speed for driving scene simulation.

Figures (10)

• Figure 1: An illustration of the proposed approach. UniGaussian reconstructs 3D driving scenes by learning unified 3D Gaussian representations from multiple input sources. It achieves holistic driving scene understanding and models multiple sensors (pinhole cameras and fisheye cameras) and modalities (semantic, normal, depth, and optional LiDAR point clouds).
• Figure 2: The framework of our UniGaussian approach to driving scene reconstruction with multiple camera models. Our approach achieves holistic driving scene understanding by modeling multiple sensors and modalities.
• Figure 3: Illustrations of our 3DGS rendering with fisheye cameras.
• Figure 4: The flowchart for analyzing geometric errors.
• Figure 5: Image zones.