AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

TL;DR

AgentFormer introduces a socio-temporal Transformer that jointly models time and social interactions for multi-agent trajectory forecasting. By using a time encoder and a novel agent-aware attention mechanism, it preserves time and agent identities within a single sequence representation. The framework employs a CVAE to jointly model latent intents across all agents, enabling diverse, socially-aware future trajectories, with a trajectory sampler enhancing sample diversity. Experiments on ETH/UCY and nuScenes show state-of-the-art performance and robust ablations demonstrate the importance of joint modeling, agent-aware attention, and latent intent sharing. This approach advances autonomous systems by producing more accurate and realistic multi-agent forecasts under uncertainty.

Abstract

Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior. Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. To this end, we propose a new Transformer, AgentFormer, that jointly models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. Our method substantially improves the state of the art on well-established pedestrian and autonomous driving datasets.

Figures (7)

• Figure 1: Different from standard approaches that model multi-agent trajectories in the time and social dimensions separately, our AgentFormer allows for joint modeling of the time and social dimensions while preserving time and agent information.
• Figure 2: Illustration of agent-aware attention. The mask $\mathbf{M}$ allows the attention weights in $\mathbf{A}$ to be computed differently based on whether the $i$-th query and $j$-th key belong to the same agent.
• Figure 3: Overview of our AgentFormer-based multi-agent trajectory prediction framework.
• Figure 4: (a,c,d) Three samples of forecasted multi-agent futures (green) via our method, which exhibit social behaviors like following (A3 & A4) and collision avoidance (A1 & A2 in (a), A2 & A3 in (c)). (b) Attention visualization for sample 1. When predicting the target (red), the model pays more attention (darker color) to key timesteps (turning point) of adjacent agents and spreads out attention to the target's past timesteps to reason about dynamics.
• Figure 5: Our method can naturally handle a time-varying number of agents because of the flexible sequence representation of multi-agent trajectories. We can simply remove the trajectory features of missing agents at each timestep from the sequence. The mask $\mathbf{M}$ of the example sequence (when applying self-attention) is computed based on the agreement of agent identity between each query and key.