VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting

TL;DR

VR-Drive addresses the critical problem of viewpoint robustness in end-to-end autonomous driving by jointly learning $3D$ scene reconstruction with feed-forward $3D$ Gaussian Splatting, enabling planning-aware novel-view synthesis from sparse data. It introduces a viewpoint-mixed memory bank and a viewpoint-consistent distillation mechanism to align representations across seen and unseen camera viewpoints, and performs training-time augmentation without additional annotations. The approach is evaluated on a new viewpoint-variation benchmark and on CARLA Town05-Nov, demonstrating strong robustness to unseen rigs and improved planning under viewpoint shifts, outperforming several prior E2E baselines. This work advances scalable, install-ready E2E-AD systems across diverse vehicle configurations by mitigating view-induced noise and enhancing 3D scene understanding.

Abstract

End-to-end autonomous driving (E2E-AD) has emerged as a promising paradigm that unifies perception, prediction, and planning into a holistic, data-driven framework. However, achieving robustness to varying camera viewpoints, a common real-world challenge due to diverse vehicle configurations, remains an open problem. In this work, we propose VR-Drive, a novel E2E-AD framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction as an auxiliary task to enable planning-aware view synthesis. Unlike prior scene-specific synthesis approaches, VR-Drive adopts a feed-forward inference strategy that supports online training-time augmentation from sparse views without additional annotations. To further improve viewpoint consistency, we introduce a viewpoint-mixed memory bank that facilitates temporal interaction across multiple viewpoints and a viewpoint-consistent distillation strategy that transfers knowledge from original to synthesized views. Trained in a fully end-to-end manner, VR-Drive effectively mitigates synthesis-induced noise and improves planning under viewpoint shifts. In addition, we release a new benchmark dataset to evaluate E2E-AD performance under novel camera viewpoints, enabling comprehensive analysis. Our results demonstrate that VR-Drive is a scalable and robust solution for the real-world deployment of end-to-end autonomous driving systems.

Figures (4)

• Figure 1: Example scenario where surrounding vehicles have stopped at a traffic signal. In the original training view, both our VR-Drive and DiffusionDrive E2E_Diffusiondrive perform well in perceiving nearby vehicles and planning. However, with a lowered camera height, DiffusionDrive fails to detect surrounding vehicles, leading to a trajectory that collides with the front vehicle, posing a safety risk. In contrast, VR-Drive maintains accurate perception (except for those occluded due to the lowered camera height) and plans trajectories as effectively as in the original view.
• Figure 2: Overall framework of VR-Drive. Our overall framework consists of three main components, as follows: (1) original-view learning, (2) novel-view learning, and (3) perception-planning learning. For novel-view learning, the perception-planning head is randomly assigned to either the original or a novel view during training, allowing the model to generalize across different viewpoints.
• Figure 3: Illustration of the perception pipeline. VR-Drive includes two complementary techniques to ensure consistent feature representations across camera viewpoints: Viewpoint-Mixed Memory Bank and Viewpoint-Consistent Distillation.
• Figure 4: Variant camera viewpoints at test time, differing from the original training distribution.