Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine

Yifei Chen, Ross Greer

TL;DR

The paper addresses the challenge of mining rare, safety-critical autonomous driving scenarios from enormous logs. It introduces SMc2f, a coarse-to-fine pipeline that first uses CLIP-based vision–language filtering on raw RGB data, then leverages a knowledge-base to provide few-shot, robust prompting for LLM-based code generation, and finally applies a text–trajectory contrastive matcher to finely rank candidate trajectories. The approach yields significant improvements over the RefAV baseline on the Argoverse 2 Scenario Mining benchmark, boosting metrics such as HOTA-T, HOTA, TS-F1, and Log-F1, while also delivering notable efficiency gains across multiple LLM backends. The work demonstrates a practical path toward scalable, simulator-ready safety validation for autonomous robots by tightly integrating multimodal perception, retrieval-augmented generation, and contrastive cross-modal learning.

Abstract

The safety validation of autonomous robotic vehicles hinges on systematically testing their planning and control stacks against rare, safety-critical scenarios. Mining these long-tail events from massive real-world driving logs is therefore a critical step in the robotic development lifecycle. The goal of the Scenario Mining task is to retrieve useful information to enable targeted re-simulation, regression testing, and failure analysis of the robot's decision-making algorithms. RefAV, introduced by the Argoverse team, is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios described in natural language. However, this process performs retrieval on trajectory labels, ignoring the direct connection between natural language and raw RGB images, which runs counter to the intuition of video retrieval; it also depends on the quality of upstream 3D object detection and tracking. Further, inaccuracies in trajectory data lead to inaccuracies in downstream spatial and temporal localization. To address these issues, we propose Robust Scenario Mining for Robotic Autonomy from Coarse to Fine (SMc2f), a coarse-to-fine pipeline that employs vision-language models (VLMs) for coarse image-text filtering, builds a database of successful mining cases on top of RefAV and automatically retrieves exemplars to few-shot condition the LLM for more robust retrieval, and introduces text-trajectory contrastive learning to pull matched pairs together and push mismatched pairs apart in a shared embedding space, yielding a fine-grained matcher that refines the LLM's candidate trajectories. Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.

SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine

TL;DR

The paper addresses the challenge of mining rare, safety-critical autonomous driving scenarios from enormous logs. It introduces SMc2f, a coarse-to-fine pipeline that first uses CLIP-based vision–language filtering on raw RGB data, then leverages a knowledge-base to provide few-shot, robust prompting for LLM-based code generation, and finally applies a text–trajectory contrastive matcher to finely rank candidate trajectories. The approach yields significant improvements over the RefAV baseline on the Argoverse 2 Scenario Mining benchmark, boosting metrics such as HOTA-T, HOTA, TS-F1, and Log-F1, while also delivering notable efficiency gains across multiple LLM backends. The work demonstrates a practical path toward scalable, simulator-ready safety validation for autonomous robots by tightly integrating multimodal perception, retrieval-augmented generation, and contrastive cross-modal learning.

Abstract

The safety validation of autonomous robotic vehicles hinges on systematically testing their planning and control stacks against rare, safety-critical scenarios. Mining these long-tail events from massive real-world driving logs is therefore a critical step in the robotic development lifecycle. The goal of the Scenario Mining task is to retrieve useful information to enable targeted re-simulation, regression testing, and failure analysis of the robot's decision-making algorithms. RefAV, introduced by the Argoverse team, is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios described in natural language. However, this process performs retrieval on trajectory labels, ignoring the direct connection between natural language and raw RGB images, which runs counter to the intuition of video retrieval; it also depends on the quality of upstream 3D object detection and tracking. Further, inaccuracies in trajectory data lead to inaccuracies in downstream spatial and temporal localization. To address these issues, we propose Robust Scenario Mining for Robotic Autonomy from Coarse to Fine (SMc2f), a coarse-to-fine pipeline that employs vision-language models (VLMs) for coarse image-text filtering, builds a database of successful mining cases on top of RefAV and automatically retrieves exemplars to few-shot condition the LLM for more robust retrieval, and introduces text-trajectory contrastive learning to pull matched pairs together and push mismatched pairs apart in a shared embedding space, yielding a fine-grained matcher that refines the LLM's candidate trajectories. Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.
Paper Structure (15 sections, 8 equations, 3 figures, 3 tables)

This paper contains 15 sections, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video retrieval
  4. Scenario mining
  5. Method
  6. CLIP-based Coarse Filtering
  7. Scenario Mining Knowledge Base
  8. Few-Shot Prompting for Robust Code Generation
  9. Fine Filtering Based on Contrastive Learning Matcher
  10. EXPERIMENTS
  11. Dataset and Evaluation Metrics
  12. Implementation Details
  13. Results
  14. Ablation Study
  15. Conclusion

Figures (3)

  • Figure 1: Overview of the RefAV baseline framework. A natural language query is processed by a Large Language Model (LLM) to synthesize an executable Python script. This script then filters the full set of trajectory data ("All Tracks") to retrieve specific segments that match the scenario description ("Mined Tracks").
  • Figure 2: The framework of SMc2f. The boxes in video frames represent randomly sampled frames within the sliding window; the red dashed lines indicate training, and the green dashed lines indicate inference.
  • Figure 3: Few-shot prompting instructions provided to LLMs.