Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models

TL;DR

PnF addresses robustness gaps in autonomous driving motion forecasting by integrating zero-shot multimodal LLM reasoning into modular prediction stacks. It introduces a Visual Semantic Analyzer and a Driving Scene Categorizer that convert language-driven scene understanding into learnable embeddings, which are fused into Transformer-based predictors via a lightweight information-gain mechanism, all without fine-tuning the LLMs. Evaluations on WOMD and nuScenes show consistent improvements across two baselines and multiple metrics, demonstrating enhanced handling of long-tail and challenging scenarios with minimal overhead. The approach is practical for real-world deployment due to its plug-and-play nature and reliance on zero-shot LLM inferences, preserving the broad generalization capabilities of LLMs.

Abstract

Current autonomous driving systems rely on specialized models for perceiving and predicting motion, which demonstrate reliable performance in standard conditions. However, generalizing cost-effectively to diverse real-world scenarios remains a significant challenge. To address this, we propose Plug-and-Forecast (PnF), a plug-and-play approach that augments existing motion forecasting models with multimodal large language models (MLLMs). PnF builds on the insight that natural language provides a more effective way to describe and handle complex scenarios, enabling quick adaptation to targeted behaviors. We design prompts to extract structured scene understanding from MLLMs and distill this information into learnable embeddings to augment existing behavior prediction models. Our method leverages the zero-shot reasoning capabilities of MLLMs to achieve significant improvements in motion prediction performance, while requiring no fine-tuning -- making it practical to adopt. We validate our approach on two state-of-the-art motion forecasting models using the Waymo Open Motion Dataset and the nuScenes Dataset, demonstrating consistent performance improvements across both benchmarks.

Figures (14)

• Figure 1: Plug-and-Forecast design. We enhance traditional autonomous driving systems by integrating MLLMs into motion forecasting models that previously relied solely on scene context from perception modules, providing comprehensive scene understanding in a zero-shot manner.
• Figure 2: Visual Prompting Example.
• Figure 3: Example Visual Semantic Analyzer (VSA) Input and Outputs.
• Figure 4: Scene Categorizer (SC) Input and Output.
• Figure 5: Architecture of PnF method. We augment a Transformer-based modular AV stack, allowing us to learn to incorporate text inputs into the stack for motion prediction. $\otimes$ stands for learned information gain operation.