Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction

TL;DR

This paper shows that the language-based model can be a powerful pedestrian trajectory predictor, and outperforms existing numerical-based predictor methods, and proposes a beam-search-based most-likely prediction and a temperature-based multimodal prediction to implement both deterministic and stochastic inferences.

Abstract

Language models have demonstrated impressive ability in context understanding and generative performance. Inspired by the recent success of language foundation models, in this paper, we propose LMTraj (Language-based Multimodal Trajectory predictor), which recasts the trajectory prediction task into a sort of question-answering problem. Departing from traditional numerical regression models, which treat the trajectory coordinate sequence as continuous signals, we consider them as discrete signals like text prompts. Specially, we first transform an input space for the trajectory coordinate into the natural language space. Here, the entire time-series trajectories of pedestrians are converted into a text prompt, and scene images are described as text information through image captioning. The transformed numerical and image data are then wrapped into the question-answering template for use in a language model. Next, to guide the language model in understanding and reasoning high-level knowledge, such as scene context and social relationships between pedestrians, we introduce an auxiliary multi-task question and answering. We then train a numerical tokenizer with the prompt data. We encourage the tokenizer to separate the integer and decimal parts well, and leverage it to capture correlations between the consecutive numbers in the language model. Lastly, we train the language model using the numerical tokenizer and all of the question-answer prompts. Here, we propose a beam-search-based most-likely prediction and a temperature-based multimodal prediction to implement both deterministic and stochastic inferences. Applying our LMTraj, we show that the language-based model can be a powerful pedestrian trajectory predictor, and outperforms existing numerical-based predictor methods. Code is publicly available at https://github.com/inhwanbae/LMTrajectory .

Figures (5)

• Figure 1: Traditional vs. Our language-based trajectory prediction, LMTraj. Given each observation data, (a) traditional predictors directly use the numerical values; (b) the proposed method converts the raw trajectory data to the linguistic prompt, and then captures reasoning social relations to predict a socially acceptable future with the question-answering template.
• Figure 2: Comparison of the text-pretrained tokenizer and our numeric data-optimized tokenizer. Under brackets with yellow or white highlight colors indicate that the corresponding letters have been tokenized. The green color highlights that the token contains an integer with 6.
• Figure 3: Visualization of prediction results on (a) zero-shot and two supervised trajectory prediction benchmarks: (b) deterministic and (c) stochastic approach. To aid visualization for the stochastic approach, we report one best trajectory of $K=20$ samples each.
• Figure 4: Visualization of the social reasoning using observed paths and the corresponding trajectory prediction results.
• Figure 5: Visualization of the most-likely and multimodal trajectory generation capability of our LMTraj- SUP ($\tau$: temperature).