MTR-VP: Towards End-to-End Trajectory Planning through Context-Driven Image Encoding and Multiple Trajectory Prediction

TL;DR

This work introduces MTR-VP, a vision-first adaptation of the Motion Transformer for end-to-end trajectory planning. By encoding scene context from stitched camera panoramas via ViT and fusing it with past motion through cross-attention, the model generates multiple plausible future trajectories conditioned on routing intent. Experiments on the Waymo End-to-End Driving Dataset reveal that while multi-trajectory prediction improves planning performance, fusing visual features with kinetic history remains challenging, and foundation-model embeddings did not consistently improve results. The study highlights the value of generating multiple futures and points to future directions in more effective vision-motion fusion and richer intent priors.

Abstract

We present a method for trajectory planning for autonomous driving, learning image-based context embeddings that align with motion prediction frameworks and planning-based intention input. Within our method, a ViT encoder takes raw images and past kinematic state as input and is trained to produce context embeddings, drawing inspiration from those generated by the recent MTR (Motion Transformer) encoder, effectively substituting map-based features with learned visual representations. MTR provides a strong foundation for multimodal trajectory prediction by localizing agent intent and refining motion iteratively via motion query pairs; we name our approach MTR-VP (Motion Transformer for Vision-based Planning), and instead of the learnable intention queries used in the MTR decoder, we use cross attention on the intent and the context embeddings, which reflect a combination of information encoded from the driving scene and past vehicle states. We evaluate our methods on the Waymo End-to-End Driving Dataset, which requires predicting the agent's future 5-second trajectory in bird's-eye-view coordinates using prior camera images, agent pose history, and routing goals. We analyze our architecture using ablation studies, removing input images and multiple trajectory output. Our results suggest that transformer-based methods that are used to combine the visual features along with the kinetic features such as the past trajectory features are not effective at combining both modes to produce useful scene context embeddings, even when intention embeddings are augmented with foundation-model representations of scene context from CLIP and DINOv2, but that predicting a distribution over multiple futures instead of a single future trajectory boosts planning performance.

Figures (3)

• Figure 1: 3 of the 8 camera images and trajectory annotations for a random sample from the Waymo dataset. (a) Front Left View, (b) Front View, (c) Front Right View. (d) The annotated front view showing the true trajectory in red and generated candidate trajectories in black. The intent for this sample is "1", which corresponds to going straight. The combination of scene input, vehicle history, and intent is used to generate the planned trajectory.
• Figure 2: MTR-VP Model Architecture: the inputs, outputs, tensors, etc. are in blue blocks and the models are in green blocks. Camera images and past states are encoded and concatenated, then decoded to form multiple possible trajectories and an associated probability distribution (a). Cross attention is applied between an intent encoding and the scene context encoding in (b), shown in the pink box. In (c), cross attention is applied between the query embedding (generated with the intent, CLIP, and DINOv2 embeddings) and the scene context encoding, then decoded to output a single future trajectory.
• Figure 3: Predicted vs. ground-truth trajectories overlaid on each of the front-facing camera views. Red dots indicate actual trajectories, and green dots indicate model predictions. The highest-likelihood model predictions are very similar to actual trajectories taken.