SCOUT: A Lightweight Framework for Scenario Coverage Assessment in Autonomous Driving

TL;DR

SCOUT introduces a lightweight surrogate model that estimates autonomous driving scenario coverage from precomputed latent sensor representations, trained via distillation from a fine-tuned LVLM. By using latent features and a labeled LVLM teacher, SCOUT delivers near-LVLM coverage accuracy with orders-of-magnitude faster inference and far lower memory requirements, enabling on-vehicle, large-scale coverage analysis. The approach demonstrates strong agreement with human annotations, substantial efficiency gains, and robustness to class imbalance, establishing a practical pathway for scalable safety evaluation in real-world driving. The work highlights the value of leveraging perception-stack features and LVLM-informed supervision to achieve scalable, accurate scenario coverage oversight for autonomous systems.

Abstract

Assessing scenario coverage is crucial for evaluating the robustness of autonomous agents, yet existing methods rely on expensive human annotations or computationally intensive Large Vision-Language Models (LVLMs). These approaches are impractical for large-scale deployment due to cost and efficiency constraints. To address these shortcomings, we propose SCOUT (Scenario Coverage Oversight and Understanding Tool), a lightweight surrogate model designed to predict scenario coverage labels directly from an agent's latent sensor representations. SCOUT is trained through a distillation process, learning to approximate LVLM-generated coverage labels while eliminating the need for continuous LVLM inference or human annotation. By leveraging precomputed perception features, SCOUT avoids redundant computations and enables fast, scalable scenario coverage estimation. We evaluate our method across a large dataset of real-life autonomous navigation scenarios, demonstrating that it maintains high accuracy while significantly reducing computational cost. Our results show that SCOUT provides an effective and practical alternative for large-scale coverage analysis. While its performance depends on the quality of LVLM-generated training labels, SCOUT represents a major step toward efficient scenario coverage oversight in autonomous systems.

Figures (5)

• Figure 1: Overview of the scenario coverage pipeline. The distilled surrogate model, SCOUT, predicts scenario coverage labels using precomputed sensor latent representations, which are inherently consumed by the agent's navigation stack. Due to the high costs incurred, only a small subset of data is annotated by humans to obtain ground-truth labels. To scale the labeling process, an LVLM is fine-tuned and later used to generate labels for a larger dataset, augmenting the training data. SCOUT, trained as a distilled surrogate model, learns to replicate the LVLM’s labeling process, thereby enabling lightweight and scalable coverage estimation.
• Figure 2: Depictions of example conflicts hankey2016description.
• Figure 3: Extraction pipeline of scenes including conflict(s). Raw camera recordings are split the into smaller scenes ($\sim$10 seconds) if they include an interesting interaction. A human then annotates them with respect to their context.
• Figure 4: Overview of the scenario coverage label generation pipeline. A driving scene is first processed to extract visually informative frames. Each frame is then encoded, alongside the tokenized/encoded scenario coverage information, and passed through the LVLM. The output is a binary label for each conflict definition, whether they exist within the scene or not.
• Figure 5: A driving scene depicting an interaction between a motorcyclist and the ego vehicle, captured from the dashcam perspective. The white motorcycle runs a red light, cutting across the intersection and triggering a high-risk encounter. Frames progress from left to right, starting at the top left and ending at the bottom right.