Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learning

Siddarth Singh, Omayma Mahjoub, Ruan de Kock, Wiem Khlifi, Abidine Vall, Kale-ab Tessera, Arnu Pretorius

TL;DR

The paper addresses the need for robust evaluation standards in deep cooperative MARL by updating an open evaluation-database with 29 papers from 2022 and comparing them to historical trends. It reveals that many longstanding issues—such as lack of uncertainty quantification, incomplete reporting, and limited algorithmic diversity—persist, though there are signs of progress with tougher SMAC-v1 scenarios and more thorough ablations. It also highlights positive developments like SMAC-v2 to combat overfitting, the use of JAX-based frameworks to improve reproducibility, and growing attention to explainability in MARL. Taken together, the work provides a practical audit of field health, underscoring the need for proactive replication practices and standardized evaluation to maintain trust as MARL research expands.

Abstract

Establishing sound experimental standards and rigour is important in any growing field of research. Deep Multi-Agent Reinforcement Learning (MARL) is one such nascent field. Although exciting progress has been made, MARL has recently come under scrutiny for replicability issues and a lack of standardised evaluation methodology, specifically in the cooperative setting. Although protocols have been proposed to help alleviate the issue, it remains important to actively monitor the health of the field. In this work, we extend the database of evaluation methodology previously published by containing meta-data on MARL publications from top-rated conferences and compare the findings extracted from this updated database to the trends identified in their work. Our analysis shows that many of the worrying trends in performance reporting remain. This includes the omission of uncertainty quantification, not reporting all relevant evaluation details and a narrowing of algorithmic development classes. Promisingly, we do observe a trend towards more difficult scenarios in SMAC-v1, which if continued into SMAC-v2 will encourage novel algorithmic development. Our data indicate that replicability needs to be approached more proactively by the MARL community to ensure trust in the field as we move towards exciting new frontiers.

How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learning

TL;DR

The paper addresses the need for robust evaluation standards in deep cooperative MARL by updating an open evaluation-database with 29 papers from 2022 and comparing them to historical trends. It reveals that many longstanding issues—such as lack of uncertainty quantification, incomplete reporting, and limited algorithmic diversity—persist, though there are signs of progress with tougher SMAC-v1 scenarios and more thorough ablations. It also highlights positive developments like SMAC-v2 to combat overfitting, the use of JAX-based frameworks to improve reproducibility, and growing attention to explainability in MARL. Taken together, the work provides a practical audit of field health, underscoring the need for proactive replication practices and standardized evaluation to maintain trust as MARL research expands.

Abstract

Establishing sound experimental standards and rigour is important in any growing field of research. Deep Multi-Agent Reinforcement Learning (MARL) is one such nascent field. Although exciting progress has been made, MARL has recently come under scrutiny for replicability issues and a lack of standardised evaluation methodology, specifically in the cooperative setting. Although protocols have been proposed to help alleviate the issue, it remains important to actively monitor the health of the field. In this work, we extend the database of evaluation methodology previously published by containing meta-data on MARL publications from top-rated conferences and compare the findings extracted from this updated database to the trends identified in their work. Our analysis shows that many of the worrying trends in performance reporting remain. This includes the omission of uncertainty quantification, not reporting all relevant evaluation details and a narrowing of algorithmic development classes. Promisingly, we do observe a trend towards more difficult scenarios in SMAC-v1, which if continued into SMAC-v2 will encourage novel algorithmic development. Our data indicate that replicability needs to be approached more proactively by the MARL community to ensure trust in the field as we move towards exciting new frontiers.
Paper Structure (5 sections, 6 figures)

This paper contains 5 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Historical trends vs new data
  4. Recent positive developments
  5. Conclusion

Figures (6)

  • Figure 1: Comparison between occurrences of algorithms from past years against the most recent year of full data.
  • Figure 2: Comparison between the performance of the Qmix algorithm from 2016-2021 against 2022.(a) Historic performance spread for Qmix in SMAC (2016-2021). (b) Recent performance spread for Qmix in SMAC (2022).
  • Figure 3: Historical tendencies for reporting evaluation runs and uncertainty metrics from 2016 to 2022(a) Tendencies for reporting evaluation runs (2016-2022). (b) Tendencies for reporting uncertainty metrics (2016-2022).
  • Figure 4: Historical tendencies for reporting performance aggregation and variance from 2016 to 2022(a) Historical aggregation usage (2016-2021). (b) Recent aggregation usage (2022). (c) Historical measure of spread (2016-2021). (d) Recent measure of spread (2022).
  • Figure 5: Historic environment usage from 2016 to 2022
  • ...and 1 more figures