Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

K-Gen: A Multimodal Language-Conditioned Approach for Interpretable Keypoint-Guided Trajectory Generation

Mingxuan Mu, Guo Yang, Lei Chen, Ping Wu, Jianxun Cui

TL;DR

K-Gen is proposed, an interpretable keypoint-guided multimodal framework that leverages Multimodal Large Language Models to unify rasterized BEV map inputs with textual scene descriptions and outperforms existing baselines.

Abstract

Generating realistic and diverse trajectories is a critical challenge in autonomous driving simulation. While Large Language Models (LLMs) show promise, existing methods often rely on structured data like vectorized maps, which fail to capture the rich, unstructured visual context of a scene. To address this, we propose K-Gen, an interpretable keypoint-guided multimodal framework that leverages Multimodal Large Language Models (MLLMs) to unify rasterized BEV map inputs with textual scene descriptions. Instead of directly predicting full trajectories, K-Gen generates interpretable keypoints along with reasoning that reflects agent intentions, which are subsequently refined into accurate trajectories by a refinement module. To further enhance keypoint generation, we apply T-DAPO, a trajectory-aware reinforcement fine-tuning algorithm. Experiments on WOMD and nuPlan demonstrate that K-Gen outperforms existing baselines, highlighting the effectiveness of combining multimodal reasoning with keypoint-guided trajectory generation.

K-Gen: A Multimodal Language-Conditioned Approach for Interpretable Keypoint-Guided Trajectory Generation

TL;DR

K-Gen is proposed, an interpretable keypoint-guided multimodal framework that leverages Multimodal Large Language Models to unify rasterized BEV map inputs with textual scene descriptions and outperforms existing baselines.

Abstract

Generating realistic and diverse trajectories is a critical challenge in autonomous driving simulation. While Large Language Models (LLMs) show promise, existing methods often rely on structured data like vectorized maps, which fail to capture the rich, unstructured visual context of a scene. To address this, we propose K-Gen, an interpretable keypoint-guided multimodal framework that leverages Multimodal Large Language Models (MLLMs) to unify rasterized BEV map inputs with textual scene descriptions. Instead of directly predicting full trajectories, K-Gen generates interpretable keypoints along with reasoning that reflects agent intentions, which are subsequently refined into accurate trajectories by a refinement module. To further enhance keypoint generation, we apply T-DAPO, a trajectory-aware reinforcement fine-tuning algorithm. Experiments on WOMD and nuPlan demonstrate that K-Gen outperforms existing baselines, highlighting the effectiveness of combining multimodal reasoning with keypoint-guided trajectory generation.
Paper Structure (16 sections, 6 equations, 3 figures, 3 tables)

This paper contains 16 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Trajectory Generation
  4. Reinforcement Fine-Tuning
  5. Method
  6. Overview
  7. Data Preprocessing Pipeline
  8. SFT-based Warm-up
  9. RFT-based Enhancement
  10. TrajRefiner
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Qualitative Analysis
  15. Ablation Study
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overall workflow of the proposed multimodal trajectory generation framework K-Gen. The MLLM takes multimodal scene data (map images and textual inputs) as input, produces both reasoning and sparse keypoints, and refines them into complete trajectories.
  • Figure 2: Overall framework of K-Gen. The framework is composed of two main components: an MLLM for reasoning and keypoint generation, and a Transformer-based module for trajectory refinement. For the MLLM, the training pipeline includes two stages: SFT with CoT outputs, followed by reinforcement RFT using the T-DAPO algorithm.
  • Figure 3: Qualitative results and attention visualization across diverse scenarios (Case 1: Intersection, Case 2: Merging, Case 3: Curved Lane). Columns (a)-(c) show the comparison between the ground truth and trajectories generated by the baseline and our TrajRefiner. Column (d) visualizes the MLLM's attention heatmaps, where warmer colors indicate higher focus on safety-critical regions and interacting agents.