Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting

TL;DR

This work introduces Collaborative Uncertainty (CU) to quantify the uncertainty arising from interaction modules in multi-agent, multi-modal trajectory forecasting. It presents a CU-aware regression framework with a permutation-equivariant uncertainty estimator that models the full joint distribution $p(\mathbf{Y}|\mathbf{X})$ via mean $\mu$ and covariance $\Sigma$, and provides a Laplace-based special case for tractable training. A CU-aware forecasting system is proposed, featuring a CU-based selection module that ranks multi-modal predictions by estimated uncertainty, enabling uncertainty-guided selection without extra confidence nets. Extensive experiments on synthetic data and two large-scale benchmarks show improved distribution estimation and forecasting performance, and reveal a positive link between future stochasticity and prediction uncertainty, as well as a clear role for interaction-induced collaborative uncertainty.

Abstract

In multi-modal multi-agent trajectory forecasting, two major challenges have not been fully tackled: 1) how to measure the uncertainty brought by the interaction module that causes correlations among the predicted trajectories of multiple agents; 2) how to rank the multiple predictions and select the optimal predicted trajectory. In order to handle these challenges, this work first proposes a novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from interaction modules. Then we build a general CU-aware regression framework with an original permutation-equivariant uncertainty estimator to do both tasks of regression and uncertainty estimation. Further, we apply the proposed framework to current SOTA multi-agent multi-modal forecasting systems as a plugin module, which enables the SOTA systems to 1) estimate the uncertainty in the multi-agent multi-modal trajectory forecasting task; 2) rank the multiple predictions and select the optimal one based on the estimated uncertainty. We conduct extensive experiments on a synthetic dataset and two public large-scale multi-agent trajectory forecasting benchmarks. Experiments show that: 1) on the synthetic dataset, the CU-aware regression framework allows the model to appropriately approximate the ground-truth Laplace distribution; 2) on the multi-agent trajectory forecasting benchmarks, the CU-aware regression framework steadily helps SOTA systems improve their performances. Specially, the proposed framework helps VectorNet improve by 262 cm regarding the Final Displacement Error of the chosen optimal prediction on the nuScenes dataset; 3) for multi-agent multi-modal trajectory forecasting systems, prediction uncertainty is positively correlated with future stochasticity; and 4) the estimated CU values are highly related to the interactive information among agents.

Figures (12)

• Figure 1: Uncertainty estimation in multi-agent multi-modal trajectory forecasting systems. (a) A typical pipeline of an encoder in multi-agent multi-modal trajectory forecasting systems. (b), (c) respectively illustrate the decoder pipeline of previous methods and our method. Previous methods output the predicted trajectory $\widehat{\mu}_{i}$ and individual uncertainty $\sigma_i$ while our method additionally outputs collaborative uncertainty$\sigma_{ij}$.
• Figure 2: Graphical models for deep learning networks in a three-agent trajectory forecasting setting: (a) represents the model that predicts the trajectory of each agent independently; (b) shows the model that explicitly captures the interactions among multiple agents (e.g., the model containing the graph-message-passing process). $\mathbf{x}_{i}$ is the observed trajectory of the $i$-th agent; $\mathbf{h}_{i}$ and $\mathbf{y}_{i}$ are its corresponding hidden feature and future trajectory respectively.
• Figure 3: Probabilistic formulation of the CU-aware regression framework. We first choose a proper probability density function for the predictive distribution based on the given dataset. Then we design a regression model $\mathcal{F}_{\boldsymbol{w}}(\cdot)$ to estimate the parameters of the chosen probability density function. Finally, on the basis of the chosen probability density function, we derive a loss function for the regression model $\mathcal{F}_{\boldsymbol{w}}(\cdot)$.
• Figure 4: Permutation-equivariant uncertainty estimator. This uncertainty estimator utilizes the input feature $\mathcal{E}$ to estimate the covariance $\Sigma_{\boldsymbol{w}}(\mathop{\mathrm{X}}\nolimits)$. $\mathbf{e}_{i}$ is the feature generated on the basis of $\mathbf{x}_{i}$ and $f_{\boldsymbol{w}}(\cdot)$ is a permutation-equivariant neural network (PE-NN). $f_{\boldsymbol{w}}(\cdot)$ aims at projecting the feature $\mathcal{E}$ to $f_{\boldsymbol{w}}(\mathcal{E})$ that is in a feature space where the individual uncertainty of $\mathbf{x}_i$ can be modeled by $\langle \mathbf{e}'_{i}$,$\mathbf{e}'_{i}\rangle$ and the collaborative uncertainty of $\mathbf{x}_i$ and $\mathbf{x}_j$ can be modeled by $\langle \mathbf{e}'_{i}$,$\mathbf{e}'_{j}\rangle$. $\tau$ is a positive real number and $I$ is an identity matrix.
• Figure 5: CU-aware multi-agent multi-modal trajectory forecasting system. This system consists of two modules: the CU-aware forecasting module and the CU-based selection module. The CU-aware forecasting module generating multi-modal predictions with the corresponding uncertainty. The CU-based selection module selects the optimal prediction generated by the forecasting module based on the estimated uncertainty. The system is trained in an end-to-end way with our proposed loss function (\ref{['eq:lformft']}).

Theorems & Definitions (5)

• Theorem 1
• Proof
• Theorem 2
• Proof
• Proof