Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving

Xiaoji Zheng, Lixiu Wu, Zhijie Yan, Yuanrong Tang, Hao Zhao, Chen Zhong, Bokui Chen, Jiangtao Gong

TL;DR

This work addresses the gap in motion prediction by enriching traffic-context understanding with large language models. It introduces Transportation Context Map prompts and structured outputs (intention, affordance, scenario) from GPT4-V, then fuses this context into a Motion Transformer via cross-attention. The approach improves prediction accuracy on the Waymo Open Motion Dataset and demonstrates a scalable, cost-efficient deployment using partial LLM augmentation. The results suggest that BEV-like context prompts enable LLMs to capture global traffic semantics that augment traditional trajectory forecasting, with potential for broader adoption in autonomous driving pipelines.

Abstract

Motion prediction is among the most fundamental tasks in autonomous driving. Traditional methods of motion forecasting primarily encode vector information of maps and historical trajectory data of traffic participants, lacking a comprehensive understanding of overall traffic semantics, which in turn affects the performance of prediction tasks. In this paper, we utilized Large Language Models (LLMs) to enhance the global traffic context understanding for motion prediction tasks. We first conducted systematic prompt engineering, visualizing complex traffic environments and historical trajectory information of traffic participants into image prompts -- Transportation Context Map (TC-Map), accompanied by corresponding text prompts. Through this approach, we obtained rich traffic context information from the LLM. By integrating this information into the motion prediction model, we demonstrate that such context can enhance the accuracy of motion predictions. Furthermore, considering the cost associated with LLMs, we propose a cost-effective deployment strategy: enhancing the accuracy of motion prediction tasks at scale with 0.7\% LLM-augmented datasets. Our research offers valuable insights into enhancing the understanding of traffic scenes of LLMs and the motion prediction performance of autonomous driving. The source code is available at \url{https://github.com/AIR-DISCOVER/LLM-Augmented-MTR} and \url{https://aistudio.baidu.com/projectdetail/7809548}.

Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving

TL;DR

This work addresses the gap in motion prediction by enriching traffic-context understanding with large language models. It introduces Transportation Context Map prompts and structured outputs (intention, affordance, scenario) from GPT4-V, then fuses this context into a Motion Transformer via cross-attention. The approach improves prediction accuracy on the Waymo Open Motion Dataset and demonstrates a scalable, cost-efficient deployment using partial LLM augmentation. The results suggest that BEV-like context prompts enable LLMs to capture global traffic semantics that augment traditional trajectory forecasting, with potential for broader adoption in autonomous driving pipelines.

Abstract

Motion prediction is among the most fundamental tasks in autonomous driving. Traditional methods of motion forecasting primarily encode vector information of maps and historical trajectory data of traffic participants, lacking a comprehensive understanding of overall traffic semantics, which in turn affects the performance of prediction tasks. In this paper, we utilized Large Language Models (LLMs) to enhance the global traffic context understanding for motion prediction tasks. We first conducted systematic prompt engineering, visualizing complex traffic environments and historical trajectory information of traffic participants into image prompts -- Transportation Context Map (TC-Map), accompanied by corresponding text prompts. Through this approach, we obtained rich traffic context information from the LLM. By integrating this information into the motion prediction model, we demonstrate that such context can enhance the accuracy of motion predictions. Furthermore, considering the cost associated with LLMs, we propose a cost-effective deployment strategy: enhancing the accuracy of motion prediction tasks at scale with 0.7\% LLM-augmented datasets. Our research offers valuable insights into enhancing the understanding of traffic scenes of LLMs and the motion prediction performance of autonomous driving. The source code is available at \url{https://github.com/AIR-DISCOVER/LLM-Augmented-MTR} and \url{https://aistudio.baidu.com/projectdetail/7809548}.
Paper Structure (14 sections, 4 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Method
  3. Get Transportation Context Information from GPT4-V
  4. TC-Map Design Suggestions
  5. Text Prompt Design Suggestions
  6. Detailed Information Supplement
  7. Multi-task Outputs Requirement
  8. Augment Motion Prediction Models via Transportation Context Information
  9. Motion Transformer
  10. Integrate with Transportation Context Information
  11. Experiement
  12. TCGP Generated Transportation Context Information
  13. LLM Augmented Motion Prediction
  14. Conclusions

Figures (4)

  • Figure 1: Context-aware Motion Prediction Based on LLMs. We first visualize the structured information of one scenario in the motion prediction dataset, GPT4-V then understand the scenario via a visualized image and well-designed prompt. Finally, GPT4-V outputs transportation context information. This information will be used to augment traditional motion prediction algorithms.
  • Figure 2: Details of Transportation Context Generation Prompt.
  • Figure 3: Six Prompt Design Guideline for GPT4-V Understanding Motion Predication Context.
  • Figure 4: Confusion Matrix of Intention Generated from GPT4-V