KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

TL;DR

KITTI-360 provides a georegistered, large-scale 360-degree driving dataset with dense 2D/3D semantic and instance labels for static and dynamic objects. It introduces a WebGL-based 3D annotation tool and a 3D-to-2D label transfer pipeline based on a dense CRF that jointly reasons over 2D pixels and 3D points, yielding coherent multi-modal annotations. The work also establishes online benchmarks for semantic scene understanding, novel view synthesis, and semantic SLAM, and demonstrates that 3D-informed labeling can outperform 2D-only baselines while significantly reducing annotation effort. Together, these contributions facilitate cross-disciplinary research at the intersection of computer vision, graphics, and robotics toward robust autonomous driving systems.

Abstract

For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

Figures (42)

• Figure 1: KITTI-360. Our dataset contains rich sensor modalities, including a perspective stereo camera, a pair of fisheye cameras, a Velodyne and a SICK laser scanning unit which together enable 360$^\circ$ scene perception. We release comprehensive annotations including consistent semantic and instance labels for every 2D image pixel and 3D point.
• Figure 2: Georegistered poses overlaid on OpenStreetMap.
• Figure 3: Annotation Interface. Our interface consists of three main components: scene view (perspective views and 3D view), semantic label panel, and controllers.
• Figure 4: 3D-to-2D Label Transfer. (\ref{['fig:model_dense_crf_projection']}) We illustrate the 3D-to-2D projection of static and dynamic object annotations. A static 3D primitive is projected to multiple frames while a dynamic 3D object is projected only into the corresponding frame. (\ref{['fig:model_dense_crf_graph']}) Factor graph representation of our model. Note that the CRF model is defined over all pixels and visible 3D points at a single timestamp. (\ref{['fig:model_dense_crf_result']}) We show the semantic inference result at timestamp $t$.
• Figure 5: Qualitative Results on Semantic Instance Segmentation Transfer. Each subfigure shows from top-to-bottom: the input image with the projected 3D points and inferred semantic segmentation boundaries, the inferred semantic instance segmentation, as well as the confidence map of the inferred label with bright and dark colors indicating high and low confidence, respectively. See supplementary material and text for details. The first scene (1st column) contains only static objects while the others (2nd and 3rd columns) also contain dynamic objects.