Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

Julius Beerwerth, Jianye Xu, Simon Schäfer, Fynn Belderink, Bassam Alrifaee

TL;DR

This work tackles the sim-to-real transfer problem for multi-agent reinforcement learning in connected and automated vehicles by introducing a reproducible benchmark that spanssimulation, a high-fidelity digital twin, and a physical cyber-physical mobility lab. It uses SigmaRL, a MAPPO-based MARL framework with structured observations and an MPC-assisted mid-level controller, to study zero-shot deployment across three realism levels. The key contributions are (i) a tri-domain benchmark platform, (ii) baseline SigmaRL deployments without fine-tuning across all domains, and (iii) a detailed analysis revealing two degradation sources: architectural differences in control stacks and a verifiable sim-to-real gap as realism increases, with collisions serving as the most robust degradation indicator. The findings validate the CPM Lab as a practical, open-source testbed for controlled, reproducible MARL sim-to-real validation and provide insights for future improvements in handling domain and environment shifts in CAV motion planning.

Abstract

We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Automated Vehicles (CAVs). The platform, based on the Cyber-Physical Mobility Lab (CPM Lab) [1], integrates simulation, a high-fidelity digital twin, and a physical testbed, enabling structured zero-shot evaluation of MARL motion-planning policies. We demonstrate its use by deploying a SigmaRL-trained policy [2] across all three domains, revealing two complementary sources of performance degradation: architectural differences between simulation and hardware control stacks, and the sim-to-real gap induced by increasing environmental realism. The open-source setup enables systematic analysis of sim-to-real challenges in MARL under realistic, reproducible conditions.

Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

TL;DR

This work tackles the sim-to-real transfer problem for multi-agent reinforcement learning in connected and automated vehicles by introducing a reproducible benchmark that spanssimulation, a high-fidelity digital twin, and a physical cyber-physical mobility lab. It uses SigmaRL, a MAPPO-based MARL framework with structured observations and an MPC-assisted mid-level controller, to study zero-shot deployment across three realism levels. The key contributions are (i) a tri-domain benchmark platform, (ii) baseline SigmaRL deployments without fine-tuning across all domains, and (iii) a detailed analysis revealing two degradation sources: architectural differences in control stacks and a verifiable sim-to-real gap as realism increases, with collisions serving as the most robust degradation indicator. The findings validate the CPM Lab as a practical, open-source testbed for controlled, reproducible MARL sim-to-real validation and provide insights for future improvements in handling domain and environment shifts in CAV motion planning.

Abstract

We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Automated Vehicles (CAVs). The platform, based on the Cyber-Physical Mobility Lab (CPM Lab) [1], integrates simulation, a high-fidelity digital twin, and a physical testbed, enabling structured zero-shot evaluation of MARL motion-planning policies. We demonstrate its use by deploying a SigmaRL-trained policy [2] across all three domains, revealing two complementary sources of performance degradation: architectural differences between simulation and hardware control stacks, and the sim-to-real gap induced by increasing environmental realism. The open-source setup enables systematic analysis of sim-to-real challenges in MARL under realistic, reproducible conditions.
Paper Structure (18 sections, 7 equations, 2 figures, 2 tables)

This paper contains 18 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contribution of this Article
  4. Organization of this Article
  5. Overview of SigmaRL
  6. Observation Design
  7. Reward Design
  8. Action Design
  9. Cyber-Physical Mobility Lab
  10. Overview
  11. Observation
  12. Action
  13. Evaluation
  14. Training in SigmaRL
  15. Experimental Setup
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Illustration of how SigmaRL's policy output is applied within the control horizon and then transitioned to a rules-based approach for the remaining prediction horizon, yielding a complete trajectory for each agent $i$ in the scene.
  • Figure 2: Example initial setup and representative trajectories for one configuration. One representative run per environment is shown, using the same initial positions across all environments; the run was selected from the digital twin as the one with centerline deviation closest to that environment's mean. Trajectories are shown over the 18s evaluation horizon; in the physical lab, a collision during this interval can further truncate the trajectory. Higher point density near the intersection occurs because vehicles slow down or briefly stop, and positions are sampled at a fixed rate. (a) Initial setup with assigned reference paths. (b) Trajectories in the SigmaRL simulation. (c) Trajectories in the digital twin. (d) Trajectories in the physical CPM Lab.