Social-Transmotion: Promptable Human Trajectory Prediction

TL;DR

Social-Transmotion addresses promptable human trajectory prediction by introducing a dual-Transformer architecture that fuses diverse visual cues, including 2D/3D pose keypoints and bounding boxes, with observed trajectories. A Cross-Modality Transformer (CMT) encodes multiple cue modalities into motion tokens, which are then integrated by a Social Transformer (ST) to model inter-agent interactions, all under a masking scheme that enables robust prediction when cues are missing. Empirical results across JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY show that 3D pose cues improve social interaction modeling and that combining multiple cues yields the best performance, while the generic masked model outperforms modality-specific baselines and remains robust to imperfect inputs. The work contributes a flexible, general framework for multimodal trajectory prediction with extensive ablations and analyses to guide cue selection in real-world deployments.

Abstract

Accurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes in the image plane, or body pose keypoints in either 2D or 3D. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Using masking technique, our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between agents based on the available visual cues. We delve into the merits of using 2D versus 3D poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and time-steps in the sequence are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY. The code is publicly available: https://github.com/vita-epfl/social-transmotion.

Figures (9)

• Figure 1: We present the task of promptable human trajectory prediction: Predict human trajectories given any available prompt such as past trajectories or body pose keypoints of all agents. Our model dynamically assesses the significance of distinct visual cues of both the primary and neighboring agents and predicts more accurate trajectories.
• Figure 2: Social-Transmotion: A Transformer-based model integrating 3D human poses and other visual cues to enhance trajectory prediction accuracy and social awareness. Cross-Modality Transformer (CMT) attends to all cues for each agent, while Social Transformer (ST) attends to all agents' representations to predict trajectories.
• Figure 3: Qualitative results showing the benefit of 3D pose input modality on trajectory predictions on the JTA dataset. Red trajectories show predictions from Social-Transmotion with only trajectory (T) as the input modality, while blue trajectories depict predictions when both of the trajectory and 3D pose (T + 3D P) input modalities are used. The ground-truth are in green and the neighboring agents are in gray. The visualization includes the last observed pose keypoints to convey walking direction and body rotation.
• Figure 4: Temporal (top) and spatial (bottom) attention maps: These maps underscore the significance of particular time-steps and body keypoints in trajectory prediction.
• Figure 5: Qualitative example of a simple turning scenario: This temporal attention map illustrates how a single key frame can be pivotal.