Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes

Riccardo Benaglia, Angelo Porrello, Pietro Buzzega, Simone Calderara, Rita Cucchiara

TL;DR

This work tackles multi-agent trajectory forecasting with a multi-modal generative model that mitigates posterior collapse by using a Vector Quantized VAE (VQ-VAE) with a context-aware, instance-based codebook. A low-rank adaptation mechanism conditions the codebook on past motion and neighbor interactions, producing a flexible yet compact latent space $e_c$ for discrete representations. A diffusion-based categorical prior then enables non-autoregressive sampling of latent codes, while a k-means centroids strategy improves multi-modal coverage and reduces noise. The approach achieves state-of-the-art results on three benchmarks (e.g., Stanford Drone, NBA, NFL) and demonstrates strong ablations, confirming the benefits of per-instance codebook customization and discrete diffusion for trajectory forecasting, with practical implications for surveillance, sports analytics, and autonomous systems.

Abstract

Trajectory forecasting is crucial for video surveillance analytics, as it enables the anticipation of future movements for a set of agents, e.g. basketball players engaged in intricate interactions with long-term intentions. Deep generative models offer a natural learning approach for trajectory forecasting, yet they encounter difficulties in achieving an optimal balance between sampling fidelity and diversity. We address this challenge by leveraging Vector Quantized Variational Autoencoders (VQ-VAEs), which utilize a discrete latent space to tackle the issue of posterior collapse. Specifically, we introduce an instance-based codebook that allows tailored latent representations for each example. In a nutshell, the rows of the codebook are dynamically adjusted to reflect contextual information (i.e., past motion patterns extracted from the observed trajectories). In this way, the discretization process gains flexibility, leading to improved reconstructions. Notably, instance-level dynamics are injected into the codebook through low-rank updates, which restrict the customization of the codebook to a lower dimension space. The resulting discrete space serves as the basis of the subsequent step, which regards the training of a diffusion-based predictive model. We show that such a two-fold framework, augmented with instance-level discretization, leads to accurate and diverse forecasts, yielding state-of-the-art performance on three established benchmarks.

Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes

TL;DR

This work tackles multi-agent trajectory forecasting with a multi-modal generative model that mitigates posterior collapse by using a Vector Quantized VAE (VQ-VAE) with a context-aware, instance-based codebook. A low-rank adaptation mechanism conditions the codebook on past motion and neighbor interactions, producing a flexible yet compact latent space for discrete representations. A diffusion-based categorical prior then enables non-autoregressive sampling of latent codes, while a k-means centroids strategy improves multi-modal coverage and reduces noise. The approach achieves state-of-the-art results on three benchmarks (e.g., Stanford Drone, NBA, NFL) and demonstrates strong ablations, confirming the benefits of per-instance codebook customization and discrete diffusion for trajectory forecasting, with practical implications for surveillance, sports analytics, and autonomous systems.

Abstract

Trajectory forecasting is crucial for video surveillance analytics, as it enables the anticipation of future movements for a set of agents, e.g. basketball players engaged in intricate interactions with long-term intentions. Deep generative models offer a natural learning approach for trajectory forecasting, yet they encounter difficulties in achieving an optimal balance between sampling fidelity and diversity. We address this challenge by leveraging Vector Quantized Variational Autoencoders (VQ-VAEs), which utilize a discrete latent space to tackle the issue of posterior collapse. Specifically, we introduce an instance-based codebook that allows tailored latent representations for each example. In a nutshell, the rows of the codebook are dynamically adjusted to reflect contextual information (i.e., past motion patterns extracted from the observed trajectories). In this way, the discretization process gains flexibility, leading to improved reconstructions. Notably, instance-level dynamics are injected into the codebook through low-rank updates, which restrict the customization of the codebook to a lower dimension space. The resulting discrete space serves as the basis of the subsequent step, which regards the training of a diffusion-based predictive model. We show that such a two-fold framework, augmented with instance-level discretization, leads to accurate and diverse forecasts, yielding state-of-the-art performance on three established benchmarks.
Paper Structure (14 sections, 9 equations, 3 figures, 5 tables)

This paper contains 14 sections, 9 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vector Quantization Models.
  4. Preliminaries
  5. Low-rank Adaptation for VQ-VAE
  6. Trajectory Forecasting with VQ-VAEs
  7. Instance-based Codebook
  8. Diffusion-based Categorical Prior
  9. Enforcing Effective Multi-modal Forecasting
  10. Experiments
  11. On the Impact of the Instance-based Codebook
  12. Comparison with SOTA Methods
  13. Qualitative Results
  14. Discussion and Conclusions

Figures (3)

  • Figure 1: Overview of our approach to trajectory prediction, based on Vector Quantization and Low-Rank adaptation of the codebook (highlighted in the purple box).
  • Figure 2: Comparison between the $K = 5$ samples obtained from a uniform sampling strategy (on the left) and the ones given as output from the proposed k-means centroids sampling strategy (on the right), starting from the same $N = 20$ initial guesses.
  • Figure 3: Qualitative comparison for three SDD scenes (one for each row of the figure) between the trajectories obtained from a VQ-VAE with a static codebook, a full rank codebook the proposed low-rank codebook (from left to right).