ESP: Extro-Spective Prediction for Long-term Behavior Reasoning in Emergency Scenarios

TL;DR

The paper tackles long-term prediction in emergency autonomous driving by introducing the ESP problem and the ESP-Dataset, which encodes rich extrospective semantic cues from the environment. It proposes a lightweight ESP encoder that can be plugged into existing predictors and a new time-aware evaluation metric, CT E, to properly assess sub-second event timing. Experimental results show ESP features consistently boost state-of-the-art backbones like TNT and MTR, with ablations highlighting the contribution of individual ESP components. The work also demonstrates the potential of integrating ESP with large language models to reason about extrospective cues, offering a path toward safer, more anticipatory autonomous driving in rare but critical scenarios.

Abstract

Emergent-scene safety is the key milestone for fully autonomous driving, and reliable on-time prediction is essential to maintain safety in emergency scenarios. However, these emergency scenarios are long-tailed and hard to collect, which restricts the system from getting reliable predictions. In this paper, we build a new dataset, which aims at the long-term prediction with the inconspicuous state variation in history for the emergency event, named the Extro-Spective Prediction (ESP) problem. Based on the proposed dataset, a flexible feature encoder for ESP is introduced to various prediction methods as a seamless plug-in, and its consistent performance improvement underscores its efficacy. Furthermore, a new metric named clamped temporal error (CTE) is proposed to give a more comprehensive evaluation of prediction performance, especially in time-sensitive emergency events of subseconds. Interestingly, as our ESP features can be described in human-readable language naturally, the application of integrating into ChatGPT also shows huge potential. The ESP-dataset and all benchmarks are released at https://dingrui-wang.github.io/ESP-Dataset/.

Figures (10)

• Figure 1: A real emergency scenario with a sedan dangerously cut-in in front of the AD truck on the highway. At the time in (a), human drivers foretell sedan's behavior by interpreting extrospective cues: 1) [observe] a high-speed accelerating (ACC) sedan approaching a Slow front-blocking truck, [predict] high potential left/right lane change and low possibility of hard brake for the sedan, 2) [observe] left lane of the sedan is Clear, [predict] left lane change will not happen as it can be done at anytime earlier with lower risk. 3) [observe] an Off-ramp exit in about 200 meters, [predict] likely to force cut-in into far-right lane to catch exit. Note the sedan did exit the highway as expected in this case. The MTR method predicts the behavior at (c). While the ESP encoder can absorb the extrospective cues to predict the cut-in event in advance as shown in (b).
• Figure 2: Sensor setup for the ESP data collection platform involves the Inceptio autonomous truck, which is equipped with 5 LiDARs, 7 cameras, 7 radars, and GPS.
• Figure 3: Organization of Semantic Infrastructure. The figure illustrates the organization of Semantic Infrastructure, which comprises three extrospective components: speed monitoring systems, junctions, and rare road objects.
• Figure 4: Seamless Integration of ESP Features with Motion Prediction Models. The ESP plugin seamlessly integrates with widely used encoder-decoder models such as TNT and MTR. As depicted, ESP enhances existing features through straightforward concatenation, leading to a transformative advancement in motion prediction.
• Figure 5: ESP Token Types - Representation of Ego (Ego vehicle), CIPV (Vehicle in front of ego), EV (Environmental vehicle), and TV (Target vehicle) within each time frame. Scenario types are determined based on rule-based criteria.