Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

TL;DR

TOKEN addresses long-tail challenges in autonomous driving by introducing object-centric tokenization that converts scenes into a compact set of object-level tokens, enabling LLM reasoning to guide planning. It leverages a pre-trained end-to-end driving model (PARA-Drive) as the scene tokenizer and trains an adapter + LoRA-based LLM in three stages to align representations and reasoning with planning. Across NuScenes data, TOKEN achieves substantial improvements in grounding, planning quality, and safety for long-tail events, including up to $27\%$ reduction in trajectory L2 error and $39\%$ fewer collisions. The results underscore the importance of representation and structured reasoning alignment and show that HD-map information and few-shot learning further boost performance, while noting limitations tied to tokenizer dependency and text-based motion outputs.

Abstract

The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system design. Traditional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM's reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by leveraging a traditional end-to-end driving model to produce condensed and semantically enriched representations of the scene, which are optimized for LLM planning compatibility through deliberate representation and reasoning alignment training stages. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning.

Figures (10)

• Figure 1: TOKEN is a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM's reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios.
• Figure 2: TOKEN obtains object-centric tokens from existing end-to-end autonomous driving stacks and uses a condensed and semantically-informed representation to encode the scene.
• Figure 3: End-to-End driving model (PARA-Drive) as the scene-tokenizer.
• Figure 4: Planning visualization in the 3-point U-turn scenario (zero-shot performance). TOKEN$^*$ denotes a model variant trained with additional synthetic data augmentation (few-shot performance).
• Figure 5: Planning performance visualization. The red dots in the left plot represent the identified critical objects. The middle plot visualizes the predicted motions. The right plot shows the speed plan inferred from the predicted motions. Note that TOKEN enables the ego vehicle to resume motion after a full stop.