COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving
Ali K. AlShami, Ananya Kalita, Ryan Rabinowitz, Khang Lam, Rishabh Bezbarua, Terrance Boult, Jugal Kalita
TL;DR
COOOL tackles the critical challenge of out-of-label hazards in autonomous driving by introducing a 200-video dashcam benchmark annotated for both known and novel hazards. The dataset serves as an evaluation-only platform to probe novelty-adjacent tasks across anomaly detection, open-set recognition, open vocabulary, and domain adaptation, and is complemented by a baseline solution and a macro-accuracy evaluation framework for hazard detection, naming, and driver reaction. Its rich, high-definition annotations include rare hazards and a substantial portion of low-resolution cases, highlighting gaps in current systems and driving the development of more robust perception and decision-making. Overall, COOOL aims to spur advancements in safe and adaptive autonomous driving by providing a realistic testbed for evaluating responses to unseen on-road hazards.
Abstract
As the Computer Vision community rapidly develops and advances algorithms for autonomous driving systems, the goal of safer and more efficient autonomous transportation is becoming increasingly achievable. However, it is 2024, and we still do not have fully self-driving cars. One of the remaining core challenges lies in addressing the novelty problem, where self-driving systems still struggle to handle previously unseen situations on the open road. With our Challenge of Out-Of-Label (COOOL) benchmark, we introduce a novel dataset for hazard detection, offering versatile evaluation metrics applicable across various tasks, including novelty-adjacent domains such as Anomaly Detection, Open-Set Recognition, Open Vocabulary, and Domain Adaptation. COOOL comprises over 200 collections of dashcam-oriented videos, annotated by human labelers to identify objects of interest and potential driving hazards. It includes a diverse range of hazards and nuisance objects. Due to the dataset's size and data complexity, COOOL serves exclusively as an evaluation benchmark.