Future-Interactions-Aware Trajectory Prediction via Braid Theory

Abstract

To safely operate, an autonomous vehicle must know the future behavior of a potentially high number of interacting agents around it, a task often posed as multi-agent trajectory prediction. Many previous attempts to model social interactions and solve the joint prediction task either add extensive computational requirements or rely on heuristics to label multi-agent behavior types. Braid theory, in contrast, provides a powerful exact descriptor of multi-agent behavior by projecting future trajectories into braids that express how trajectories cross with each other over time; a braid then corresponds to a specific mode of coordination between the multiple agents in the future. In past work, braids have been used lightly to reason about interacting agents and restrict the attention window of predicted agents. We show that leveraging more fully the expressivity of the braid representation and using it to condition the trajectories themselves leads to even further gains in joint prediction performance, with negligible added complexity either in training or at inference time. We do so by proposing a novel auxiliary task, braid prediction, done in parallel with the trajectory prediction task. By classifying edges between agents into their correct crossing types in the braid representation, the braid prediction task is able to imbue the model with improved social awareness, which is reflected in joint predictions that more closely adhere to the actual multi-agent behavior. This simple auxiliary task allowed us to obtain significant improvements in joint metrics on three separate datasets. We show how the braid prediction task infuses the model with future intention awareness, leading to more accurate joint predictions. Code is available at github.com/caiocj1/traj-pred-braid-theory.

Figures (6)

• Figure 1: Schematics of our method. During inference, initial mode embeddings $\mathbf{m}_k$ corresponding to different behaviors are updated with scene context; the resulting final mode embeddings from the decoder are passed to both braid and trajectory prediction tasks. In braid prediction, final mode embeddings $\mathbf{m}'_{i,k}$, $\mathbf{m}'_{j,k}$ that started from the same initial embedding $\mathbf{m}_k$ are fused for each pair of agents $(i,j)$ into edge features $\mathbf{f}_{i \rightarrow j,k}$ that are classified into their correct class from the corresponding braid representation. During training, the cross-entropy loss $\mathcal{L}_{\text{braid}}$ is applied to the mode with least joint displacement error.
• Figure 2: On the left, example of composition operation with generators, and of a braid $b \in B_6$. Notice that $b$ can be described as a composition of braid generators resulting from crossings in the $xt$ plane, and that the kind of crossing is determined by relative depth in the $y$ axis at crossing points. On the right, example of braid computation from traffic scene's future ground-truth trajectories. Note that, for instance, the green trajectory crossing under the red one implies agent 2 yielding to agent 3.
• Figure 3: Illustrative example of crossing labels in two separate traffic scenes with different behaviors for a pair of agents. Lines with final circles indicate future ground-truth trajectories.
• Figure 4: Decoding final mode embeddings for agents $i$ and $j$, assuming $K=2$. Final updated mode embeddings are then used in both braid and trajectory prediction tasks.
• Figure 5: Example decoding crossing probabilities $\text{softmax}(\widehat{\mathbf{ c }}_{i \rightarrow j})$ for the edge from agent $i$ to $j$, assuming $K=2$. Best mode $k^*$ is the one that minimizes the joint displacement error of $i$ and $j$.