Distilling Multi-modal Large Language Models for Autonomous Driving

TL;DR

<3-5 sentence high-level summary> This paper tackles robustness in autonomous driving under long-tail events by leveraging the world knowledge of multi-modal LLMs while preserving the efficiency of vision-based planning. It introduces DiMA, a joint training framework where a scene encoder produces BEAM tokens that feed both a vision-based planner and an MLLM, with surrogate tasks and a distillation objective enabling grounded, language-informed planning. The LLM can be discarded at inference, maintaining speed, yet can contribute through the MLLM branch for language-guided reasoning. On nuScenes, DiMA achieves state-of-the-art planning performance, notable reductions in trajectory error and collisions, and strong improvements in long-tail scenarios, highlighting its practical impact for robust autonomous driving systems.

Abstract

Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.

Figures (15)

• Figure 1: Comparison of planning performance in long-tail scenarios from nuScenes: DiMA-VAD demonstrates greater robustness compared to VAD jiang2023vad in long-tail navigation scenarios such as overtaking a vehicle and performing a 3-point turn. DiMA-VAD also outperforms recent vision-planner PARA-Drive weng2024drive and LLM planner TOKEN tian2024tokenize. Notably, the 3-point turn is a zero-shot scenario that is only present in the validation set.
• Figure 2: Overview of DiMA. The input to the framework is a multi-view image sequence and a question text prompt. The vision-based end-to-end planner consists of a scene encoder and a planning transformer. The scene encoder learns structured latent representations in the form of bird's-eye-view, ego, agent, and map ($BEAM$) token embeddings and acts as a trainable tokenizer for the multi-modal large language model (MLLM). The planning transformer is trained under standard planning constraints jiang2023vadhu2023planning. The MLLM is trained for planning, visual question answering, distillation, and a series of surrogate tasks.
• Figure 3: Examples of addition and deletion in scene editing. In the top row, a car (solid red box) is added on the left of the ego-vehicle (green box). In the bottom row, a car (dashed red box) is removed from the right of the ego-vehicle. A corresponding question-answer pair is created to characterize the edit.
• Figure 4: Visualization of planning performance by the MLLM branch of DiMA-VAD-Tiny. We plot the predicted trajectory (orange-yellow) and show an example response of the language model branch to a question from the DriveLM test dataset sima2023drivelm.
• Figure 5: Overview of Surrogate tasks. Here hidden token embeddings are latent representations from the penultimate layer of LLM. These hidden token embeddings corresponding to bird's-eye-view, ego, agent, and map ($BEAM$) token embeddings are used as input as surrogate task decoder heads to perform masked reconstruction, future prediction and scene editing.