Learned Collusion
Olivier Compte
TL;DR
Problem: understanding whether exogenously defined $Q$-learning automata can sustain cooperation or collusion across diverse payoff and monitoring structures. Method: introduce $Qb$-learning, a one-dimensional bias added to the $Q$-value decision rule, and analyze the induced game over biases using the limit Quantal Response Equilibrium. Findings: $Qb$-equilibria promote persistent cooperation in the Prisoner’s Dilemma and substantial collusion in a duopoly, robust to stochastic payoffs and initial $Q$-values; biases help escape $Q$-traps and improve welfare relative to naive $Q$-learning. Significance: lenient learning rules can emerge endogenously under simple logit/best-response dynamics, challenging classic monitoring assumptions and suggesting extensions to other reinforcement-learning rules.
Abstract
Q-learning can be described as an all-purpose automaton that provides estimates (Q-values) of the continuation values associated with each available action and follows the naive policy of almost always choosing the action with highest Q-value. We consider a family of automata based on Q-values, whose policy may systematically favor some actions over others, for example through a bias that favors cooperation. We look for stable equilibrium biases, easily learned under converging logit/best-response dynamics over biases, not requiring any tacit agreement. These biases strongly foster collusion or cooperation across a rich array of payoff and monitoring structures, independently of initial Q-values.