Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Off-Policy Prediction for Multi-Agent Systems

Tom Kuipers, Renukanandan Tumu, Shuo Yang, Milad Kazemi, Rahul Mangharam, Nicola Paoletti

TL;DR

MA-COPP extends conformal off-policy prediction to multi-agent systems by constructing joint prediction regions for future trajectories under target-policy shifts affecting ego agents. It avoids exhaustive output-space enumeration by deriving a conservative max-density-ratio (max-DR) based reweighting of the calibration distribution and lifting the task to augmented sequences that include ego actions, with a multivariate time-series non-conformity score. The method provides finite-sample, distribution-free guarantees and demonstrates nominal coverage in high-dimensional settings on PettingZoo MPE and F1TENTH, outperforming standard CP baselines that ignore distribution shift and maintaining practical region sizes. This enables dependable off-policy trajectory predictions in safety-critical multi-agent domains, while highlighting trade-offs under large policy shifts where density ratios can become large or unbounded. Overall, MA-COPP broadens the applicability of CP-based uncertainty quantification to complex multi-agent dynamics with policy changes.

Abstract

Off-Policy Prediction (OPP), i.e., predicting the outcomes of a target policy using only data collected under a nominal (behavioural) policy, is a paramount problem in data-driven analysis of safety-critical systems where the deployment of a new policy may be unsafe. To achieve dependable off-policy predictions, recent work on Conformal Off-Policy Prediction (COPP) leverage the conformal prediction framework to derive prediction regions with probabilistic guarantees under the target process. Existing COPP methods can account for the distribution shifts induced by policy switching, but are limited to single-agent systems and scalar outcomes (e.g., rewards). In this work, we introduce MA-COPP, the first conformal prediction method to solve OPP problems involving multi-agent systems, deriving joint prediction regions for all agents' trajectories when one or more ego agents change their policies. Unlike the single-agent scenario, this setting introduces higher complexity as the distribution shifts affect predictions for all agents, not just the ego agents, and the prediction task involves full multi-dimensional trajectories, not just reward values. A key contribution of MA-COPP is to avoid enumeration or exhaustive search of the output space of agent trajectories, which is instead required by existing COPP methods to construct the prediction region. We achieve this by showing that an over-approximation of the true joint prediction region (JPR) can be constructed, without enumeration, from the maximum density ratio of the JPR trajectories. We evaluate the effectiveness of MA-COPP in multi-agent systems from the PettingZoo library and the F1TENTH autonomous racing environment, achieving nominal coverage in higher dimensions and various shift settings.

Conformal Off-Policy Prediction for Multi-Agent Systems

TL;DR

MA-COPP extends conformal off-policy prediction to multi-agent systems by constructing joint prediction regions for future trajectories under target-policy shifts affecting ego agents. It avoids exhaustive output-space enumeration by deriving a conservative max-density-ratio (max-DR) based reweighting of the calibration distribution and lifting the task to augmented sequences that include ego actions, with a multivariate time-series non-conformity score. The method provides finite-sample, distribution-free guarantees and demonstrates nominal coverage in high-dimensional settings on PettingZoo MPE and F1TENTH, outperforming standard CP baselines that ignore distribution shift and maintaining practical region sizes. This enables dependable off-policy trajectory predictions in safety-critical multi-agent domains, while highlighting trade-offs under large policy shifts where density ratios can become large or unbounded. Overall, MA-COPP broadens the applicability of CP-based uncertainty quantification to complex multi-agent dynamics with policy changes.

Abstract

Off-Policy Prediction (OPP), i.e., predicting the outcomes of a target policy using only data collected under a nominal (behavioural) policy, is a paramount problem in data-driven analysis of safety-critical systems where the deployment of a new policy may be unsafe. To achieve dependable off-policy predictions, recent work on Conformal Off-Policy Prediction (COPP) leverage the conformal prediction framework to derive prediction regions with probabilistic guarantees under the target process. Existing COPP methods can account for the distribution shifts induced by policy switching, but are limited to single-agent systems and scalar outcomes (e.g., rewards). In this work, we introduce MA-COPP, the first conformal prediction method to solve OPP problems involving multi-agent systems, deriving joint prediction regions for all agents' trajectories when one or more ego agents change their policies. Unlike the single-agent scenario, this setting introduces higher complexity as the distribution shifts affect predictions for all agents, not just the ego agents, and the prediction task involves full multi-dimensional trajectories, not just reward values. A key contribution of MA-COPP is to avoid enumeration or exhaustive search of the output space of agent trajectories, which is instead required by existing COPP methods to construct the prediction region. We achieve this by showing that an over-approximation of the true joint prediction region (JPR) can be constructed, without enumeration, from the maximum density ratio of the JPR trajectories. We evaluate the effectiveness of MA-COPP in multi-agent systems from the PettingZoo library and the F1TENTH autonomous racing environment, achieving nominal coverage in higher dimensions and various shift settings.
Paper Structure (20 sections, 3 theorems, 15 equations, 3 figures, 2 tables)

This paper contains 20 sections, 3 theorems, 15 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The Multi-Agent OPP Problem
  3. Notation
  4. Behavioural process
  5. Observational data
  6. Problem definition
  7. Background
  8. The MA-COPP approach
  9. Lifting of prediction task
  10. Density ratio computation
  11. Non-conformity score
  12. Construction of Prediction Regions
  13. Results
  14. Experimental Settings
  15. Case Studies
  16. ...and 5 more sections

Key Result

Proposition 1

For $\alpha\in (0,1)$ and $H<T$, let $C^+_{\alpha}$ be a JPR valid for $\mathbf{Y}_{H+1\ldots T}$, i.e., such that Let $C'_{\alpha}(\mathbf{X}_{1\ldots H}) = \Pi_{\mathbf{x}_{H+1\ldots T}}( C^+_{\alpha}(\mathbf{X}_{1\ldots H}))$ be the projection of the JPR onto components $\mathbf{x}_{H+1\ldots T}$. Then, $C'_{\alpha}$ provides a solution to Problem prob:ma-copp in that

Figures (3)

  • Figure 1: 2D visualisation of actual JPRs constructed over the position of a single ego agent in the MPE environment defined in § \ref{['sec:mpe-results']}
  • Figure 2: Overview of the MA-COPP method. A calibration distribution is first derived from behavioural data and a predictor (see § \ref{['sec:non-conformity-score']}). Density ratios are computed as described in § \ref{['sec:density-ratio']}. To construct the JPR for a given test point, we estimate the maximum DR over all the outputs that pass the CP test, and use this estimate to reweight the calibration distribution, see § \ref{['sec:construct-pred-regions']}.
  • Figure 3: $x$-axis: $\epsilon_{bias}$; Dashed red: nominal coverage level; Red: standard CP $B \rightarrow T$; Green: standard CP $T \rightarrow T$; Blue: MA-COPP with synthetic process $B^{S} \rightarrow T$; Magenta: MA-COPP with true process $B^{T} \rightarrow T$

Theorems & Definitions (9)

  • Remark 1
  • Remark 2
  • Remark 3
  • Proposition 1
  • proof
  • Proposition 2: Max-DR region
  • proof
  • Proposition 3
  • proof