Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games

TL;DR

The paper tackles mixed-motive multi-agent reinforcement learning by introducing LASE, a decentralized algorithm that balances altruism and self-interest through empathy-based gifting. LASE infers social relationships with co-players via counterfactual reasoning against a perspective-taking module, and uses a zero-sum gifting scheme to modulate reward sharing, guiding policies toward cooperative behavior while mitigating exploitation. The authors provide theoretical analysis in iterated matrix games and validate the approach across IPD and four sequential social dilemmas, showing improved cooperation, fairness, and adaptability to diverse co-players. This work offers a scalable framework for empathy-informed decision-making in decentralized multi-agent systems with practical implications for autonomous collaboration and negotiation settings.

Abstract

Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by inferred social relationships between agents, we propose LASE Learning to balance Altruism and Self-interest based on Empathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship -- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by comparing the estimated $Q$-function of current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments are performed in spatially and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.

Figures (10)

• Figure 1: Architecture of LASE. It consists of Social Relationships Inference (SRI) and Gifting. SRI conducts counterfactual reasoning to get the social relationships with co-players. The social relationship is measured by comparing the $Q$-value (estimated by the SR value network) of the current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking (PT) module. PT is provided to address the challenge of predicting co-players' policies in partially observable and decentralized environments. The Gifting module, according to the inferred social relationships, determines the amount of reward to share.
• Figure 2: The cooperation probability of LASE agents after convergence under different matrix-game parameters. The X-axis and Y-axis represent two parameters $T$ and $S$ respectively, where $T\in[0:0.02:2]$ and $S\in[-1:0.02:1]$.
• Figure 3: Graphic representations of four SSDs: (a) Coingame (5$\times$5 map), (b) Cleanup (8$\times$8 map), (c) SSH (8$\times$8 map), (d) SSG (8$\times$8 map).
• Figure 4: Results in IPD. (a) The learning path of two LASE agents. They start from the lower left and converge to the upper right of the phase diagram, where both agents cooperate with a probability around $0.93$. (b) The collective reward of LASE and baselines. Five seeds are randomly selected for the experiment. The solid line represents the mean performance, while the shaded area indicates the standard deviation.
• Figure 5: Learning curves in four SSDs. Shown is the collective reward. All the curves are plotted using 5 training runs with different random seeds, where the solid line is the mean and the shadowed area indicates the standard deviation.