Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games

TL;DR

Problem: enabling autonomous agents to learn and sustain shared normative systems within human-like institutions. Approach: norm-augmented Markov games (NMGs) that embed a normative set $N$ with priors $P^0_i(N)$ and posteriors $P^t_i(N)$, and use approximately Bayesian rule induction for norm learning, paired with model-based planning that toggles between reward-oriented and obligation-satisfying modes. Key contributions: formalization of NMGs, representation of norms as prohibitions and obligations, mean-field Bayesian updating, and empirical validation in the Melting Pot environment showing rapid norm acquisition, convergence to common norms, and intergenerational transmission. Significance: provides a scalable, cognitively plausible framework for norm-aware autonomous systems that can coordinate with human institutions and sustain cooperative welfare.

Abstract

A universal feature of human societies is the adoption of systems of rules and norms in the service of cooperative ends. How can we build learning agents that do the same, so that they may flexibly cooperate with the human institutions they are embedded in? We hypothesize that agents can achieve this by assuming there exists a shared set of norms that most others comply with while pursuing their individual desires, even if they do not know the exact content of those norms. By assuming shared norms, a newly introduced agent can infer the norms of an existing population from observations of compliance and violation. Furthermore, groups of agents can converge to a shared set of norms, even if they initially diverge in their beliefs about what the norms are. This in turn enables the stability of the normative system: since agents can bootstrap common knowledge of the norms, this leads the norms to be widely adhered to, enabling new entrants to rapidly learn those norms. We formalize this framework in the context of Markov games and demonstrate its operation in a multi-agent environment via approximately Bayesian rule induction of obligative and prohibitive norms. Using our approach, agents are able to rapidly learn and sustain a variety of cooperative institutions, including resource management norms and compensation for pro-social labor, promoting collective welfare while still allowing agents to act in their own interests.

Figures (9)

• Figure 1: Overview of our framework, summarized by (a) the (simplified) graphical model for a norm-augmented Markov game. Agents learn to comply with rule-based social norms (b), which are factored into prohibitions (e.g. P1, P2) and obligations (e.g. O1, O2, O3). Agents comply with norms while pursuing their interests via model-based planning (c), which switches between a reward-oriented mode that obeys prohibitions while collecting rewards (left), and an obligation-oriented mode that plans to satisfy a postcondition (right). Agents learn these norms via (d) approximate Bayesian inference, updating the probabilities of each potential norm based on observations of others' actions.
• Figure 2: Passive Norm Learning. Average posterior belief of the learner in each of the five norms practiced by the experienced agents, $N_\text{active}$. Most norms were acquired within $\leq 300$ steps.
• Figure 3: Social outcomes for two sets of norms $N$ practiced by the experienced agents: either all norms in $\{\textsf{P1}, \textsf{P2}, \textsf{O1}, \textsf{O2}, \textsf{O3}\}$ (Norms active) or none of them (Norms not active).
• Figure 4: Intergenerational transmission of norms. Mean norm belief averaged across all six agents across generations.
• Figure 5: Norm emergence and convergence. Geometric and arithmetic mean posterior beliefs of all agents for the top 5 norms with final belief $P^{t=300}(n)\geq \theta = 0.95$.