GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems

TL;DR

The paper addresses the challenge of brittle reward engineering in multi-agent reinforcement learning, especially under sparse rewards. It proposes GOV-REK, a governance framework that injects simple, geometry-based reward priors (governance kernels) as a governance layer to bias agent incentives, optimized via a Hyperband-like search for problem-agnostic configurations. The approach defines agent-specific and agent-agnostic kernels over state or joint-action spaces, allows kernel superimposition and mutation, and enforces PBRS-consistent normalization. Empirical results across spatial (2D/3D) and non-spatial MARL tasks show faster convergence, improved robustness, and scalable performance compared with a manually engineered MORS baseline, illustrating practical impact for designing robust MARL systems.

Abstract

For multi-agent reinforcement learning systems (MARLS), the problem formulation generally involves investing massive reward engineering effort specific to a given problem. However, this effort often cannot be translated to other problems; worse, it gets wasted when system dynamics change drastically. This problem is further exacerbated in sparse reward scenarios, where a meaningful heuristic can assist in the policy convergence task. We propose GOVerned Reward Engineering Kernels (GOV-REK), which dynamically assign reward distributions to agents in MARLS during its learning stage. We also introduce governance kernels, which exploit the underlying structure in either state or joint action space for assigning meaningful agent reward distributions. During the agent learning stage, it iteratively explores different reward distribution configurations with a Hyperband-like algorithm to learn ideal agent reward models in a problem-agnostic manner. Our experiments demonstrate that our meaningful reward priors robustly jumpstart the learning process for effectively learning different MARL problems.

Figures (10)

• Figure 1: The systematic reward configuration exploration with exponentially increasing training timestep budgets for learning.
• Figure 2: The functional description of the governance layer for the sparse package delivery cooperative MARL problem.
• Figure 3: The sequence of execution and learning steps in GMAS and GOV-REK approaches.
• Figure 4: The different ways in which different governance kernels are superimposable and adaptable as agent-specific and agent-agnostic reward distributions.
• Figure 5: The increasing periodic governance kernel configuration for the non-spatial social dilemma problems.