Remember and Recall: Associative-Memory-based Trajectory Prediction

TL;DR

The Fragmented-Memory-based Trajectory Prediction model (FMTP) inspired by the remarkable learning capabilities of humans is proposed, employing discrete representations to enhance computational efficiency by reducing information redundancy while maintaining the flexibility to utilize past experiences.

Abstract

Trajectory prediction is a pivotal component of autonomous driving systems, enabling the application of accumulated movement experience to current scenarios. Although most existing methods concentrate on learning continuous representations to gain valuable experience, they often suffer from computational inefficiencies and struggle with unfamiliar situations. To address this issue, we propose the Fragmented-Memory-based Trajectory Prediction (FMTP) model, inspired by the remarkable learning capabilities of humans, particularly their ability to leverage accumulated experience and recall relevant memories in unfamiliar situations. The FMTP model employs discrete representations to enhance computational efficiency by reducing information redundancy while maintaining the flexibility to utilize past experiences. Specifically, we design a learnable memory array by consolidating continuous trajectory representations from the training set using defined quantization operations during the training phase. This approach further eliminates redundant information while preserving essential features in discrete form. Additionally, we develop an advanced reasoning engine based on language models to deeply learn the associative rules among these discrete representations. Our method has been evaluated on various public datasets, including ETH-UCY, inD, SDD, nuScenes, Waymo, and VTL-TP. The extensive experimental results demonstrate that our approach achieves significant performance and extracts more valuable experience from past trajectories to inform the current state.

Figures (5)

• Figure 1: An illustration of the excellent adaptability of humans in dealing with various situations. Facing new environments and challenges, people can quickly extract and analyze fragmented memories from past experiences in order to find experiences that match the current situation. This ability enables humans to process information efficiently and respond rationally and adaptably.
• Figure 2: The overall framework of FMTP. In our method, the discrete representations of agent trajectories are captured through quantization operations during the trajectory reconstruction process and are stored in a specially designed memory array. Simultaneously, the encoder and decoder are also trained during this process. We utilize the trained encoder and memory array to extract the index sequences of the trajectory's discrete representations. Based on these index sequences, we employ a Transformer model to accurately model the structure and characteristics of the trajectory's constituent parts.
• Figure 3: The time cost of FMTP and MemoNet implementing one prediction on the ETH-UCY.
• Figure 4: The visualization of the predicted trajectories by FMTP compared to MemoNet, reproduced with pre-trained weights. Four columns display different motion patterns. To aid in visualization, the trajectories with the best ADE among 20 samples are reported.
• Figure 5: The visualization results of multimodal prediction for our method on ETH-UCY dataset.