Trust-based Consensus in Multi-Agent Reinforcement Learning Systems

TL;DR

This work tackles the challenge of achieving consensus in multi-agent reinforcement learning (MARL) when some agents may be unreliable. It introduces Reinforcement Learning-based Trusted Consensus (RLTC), a decentralized mechanism where each agent learns which neighbors to trust, allowing it to ignore misleading information and still converge to the correct value. Formulated as independent Q-learning across agents, RLTC demonstrates improved consensus success rates over non-trust baselines and scalability to larger networks, while generalizing to Fixed and Random failure models. The findings highlight the viability of emergent, decentralized trust as a modular component for robust coordination in real-world MARL systems. The approach has practical implications for deploying cooperative AI in environments with noise, faults, or adversarial behavior.

Abstract

An often neglected issue in multi-agent reinforcement learning (MARL) is the potential presence of unreliable agents in the environment whose deviations from expected behavior can prevent a system from accomplishing its intended tasks. In particular, consensus is a fundamental underpinning problem of cooperative distributed multi-agent systems. Consensus requires different agents, situated in a decentralized communication network, to reach an agreement out of a set of initial proposals that they put forward. Learning-based agents should adopt a protocol that allows them to reach consensus despite having one or more unreliable agents in the system. This paper investigates the problem of unreliable agents in MARL, considering consensus as a case study. Echoing established results in the distributed systems literature, our experiments show that even a moderate fraction of such agents can greatly impact the ability of reaching consensus in a networked environment. We propose Reinforcement Learning-based Trusted Consensus (RLTC), a decentralized trust mechanism, in which agents can independently decide which neighbors to communicate with. We empirically demonstrate that our trust mechanism is able to handle unreliable agents effectively, as evidenced by higher consensus success rates.

Figures (16)

• Figure 1: Communication network example with 9 agents, where nodes represent agents, arrows indicate communication links and colors are mapped to the agents' local values.
• Figure 2: Communication mechanism from the perspective of agent 5. Agent 5 receives the values from its neighbors, then updates its local value by randomly selecting from the set of received values and its own. During each timestep, this is performed simultaneously by all agents.
• Figure 3: Example of an unreliable agent 1 (square) that always ignores messages from neighbors 2 and 3 and sends 0.
• Figure 4: Communication between agent 5 and its neighbors, but agent 5 is equipped with a trust mechanism. Agent 5 trusts neighbors 4, 6 and 8 but not agent 2. It updates its local value by randomly sampling from itself and its trusted neighbors only.
• Figure 5: Communication grid example with 9 agents, in which not all agents trust each other.