SMEMO: Social Memory for Trajectory Forecasting

TL;DR

This paper tackles multimodal trajectory forecasting in crowded environments by introducing SMEMO, a Memory Augmented Neural Network with an external, trainable working memory that stores per-agent social cues. Through egocentric and social streams, a shared memory, and read/write addressing, SMEMO produces multiple diverse futures while enabling explainability via memory access patterns. The approach achieves state-of-the-art or competitive results on ETH/UCY and SDD, and demonstrates strong reasoning about social interactions on synthetic SSA through both quantitative metrics (ADE/FDE, Kendall’s tau, CEA) and qualitative analyses. The external memory framework provides a natural platform for interpreting cause–effect relations in agent behavior, which is particularly valuable for safety-critical autonomous systems.

Abstract

Effective modeling of human interactions is of utmost importance when forecasting behaviors such as future trajectories. Each individual, with its motion, influences surrounding agents since everyone obeys to social non-written rules such as collision avoidance or group following. In this paper we model such interactions, which constantly evolve through time, by looking at the problem from an algorithmic point of view, i.e. as a data manipulation task. We present a neural network based on an end-to-end trainable working memory, which acts as an external storage where information about each agent can be continuously written, updated and recalled. We show that our method is capable of learning explainable cause-effect relationships between motions of different agents, obtaining state-of-the-art results on multiple trajectory forecasting datasets.

Figures (12)

• Figure 1: SMEMO models social interaction in trajectory prediction exploiting a working memory. Past observations are encoded and stored in memory and retrieved to formulate multiple future predictions for each agent. Read/Write operations are guided by a controller. The whole model is trained end-to-end.
• Figure 2: A social context $\mathcal{S}$ with two agents $\mathbf{x}^0$ and $\mathbf{x}^1$. Past (blue) is observed and multiple futures are predicted (shades of red).
• Figure 3: Past trajectories are fed to the Egocentric stream (Blue) and a Social stream (Yellow). In the Egocentric stream, each past trajectory is embedded and then encoded with a Recurrent Motion Encoder GRU. In the Social stream, the Social Memory Module encodes past trajectories and populates the working memory, which is shared for each agent. This allows the model to create a representation for the social context. Prediction is performed by decoding the concatenation of the past encoding and the social feature.
• Figure 4: Social Memory Module Addressing. SMEMO is equipped with a shared memory $\textbf{M}_t$. The controller outputs at each timestep a feature $\gamma^{i}_{t}$ which is fed to the read/write heads to generate a memory key $\boldsymbol{\eta}$ and an addressing strength $\beta$. The key is used to find relevant memory locations in memory via cosine similarity. Access weights $\boldsymbol{\alpha}$ are then obtained by normalizing such similarities through a softmax with temperature $\beta$.
• Figure 5: Social Memory Module Reading. For each agent $i$, separate read heads perform a memory addressing to obtain $K$ social features $\sigma^i_{t,k}$ which will be fed in parallel into the decoder to generate a multimodal future prediction. The social features are then pooled together via Future Pooling and fed back to the model auto-regressively.