Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

By Fair Means or Foul: Quantifying Collusion in a Market Simulation with Deep Reinforcement Learning

Michael Schlechtinger, Damaris Kosack, Franz Krause, Heiko Paulheim

TL;DR

This paper investigates algorithmic collusion in a pricing game by deploying multi-agent deep reinforcement learning (DRL) within a scalable oligopoly simulation. It introduces a novel demand framework that blends multiple demand models via a bias parameter $\mu \in [0,1]$ and analyzes two scenarios, including a constrained-observation setting, to study convergence to supracompetitive prices. Using model-free DRL algorithms (PPO and DQN) in a discrete action space, the study shows that agents can converge to collusive-like pricing with oscillatory dynamics, even without direct inter-agent communication, and that these outcomes persist under ablations. The work discusses legal implications under EU and German competition law and argues for policy and regulatory considerations to mitigate algorithmic collusion while highlighting the broader impact on e-commerce pricing strategies.

Abstract

In the rapidly evolving landscape of eCommerce, Artificial Intelligence (AI) based pricing algorithms, particularly those utilizing Reinforcement Learning (RL), are becoming increasingly prevalent. This rise has led to an inextricable pricing situation with the potential for market collusion. Our research employs an experimental oligopoly model of repeated price competition, systematically varying the environment to cover scenarios from basic economic theory to subjective consumer demand preferences. We also introduce a novel demand framework that enables the implementation of various demand models, allowing for a weighted blending of different models. In contrast to existing research in this domain, we aim to investigate the strategies and emerging pricing patterns developed by the agents, which may lead to a collusive outcome. Furthermore, we investigate a scenario where agents cannot observe their competitors' prices. Finally, we provide a comprehensive legal analysis across all scenarios. Our findings indicate that RL-based AI agents converge to a collusive state characterized by the charging of supracompetitive prices, without necessarily requiring inter-agent communication. Implementing alternative RL algorithms, altering the number of agents or simulation settings, and restricting the scope of the agents' observation space does not significantly impact the collusive market outcome behavior.

By Fair Means or Foul: Quantifying Collusion in a Market Simulation with Deep Reinforcement Learning

TL;DR

This paper investigates algorithmic collusion in a pricing game by deploying multi-agent deep reinforcement learning (DRL) within a scalable oligopoly simulation. It introduces a novel demand framework that blends multiple demand models via a bias parameter and analyzes two scenarios, including a constrained-observation setting, to study convergence to supracompetitive prices. Using model-free DRL algorithms (PPO and DQN) in a discrete action space, the study shows that agents can converge to collusive-like pricing with oscillatory dynamics, even without direct inter-agent communication, and that these outcomes persist under ablations. The work discusses legal implications under EU and German competition law and argues for policy and regulatory considerations to mitigate algorithmic collusion while highlighting the broader impact on e-commerce pricing strategies.

Abstract

In the rapidly evolving landscape of eCommerce, Artificial Intelligence (AI) based pricing algorithms, particularly those utilizing Reinforcement Learning (RL), are becoming increasingly prevalent. This rise has led to an inextricable pricing situation with the potential for market collusion. Our research employs an experimental oligopoly model of repeated price competition, systematically varying the environment to cover scenarios from basic economic theory to subjective consumer demand preferences. We also introduce a novel demand framework that enables the implementation of various demand models, allowing for a weighted blending of different models. In contrast to existing research in this domain, we aim to investigate the strategies and emerging pricing patterns developed by the agents, which may lead to a collusive outcome. Furthermore, we investigate a scenario where agents cannot observe their competitors' prices. Finally, we provide a comprehensive legal analysis across all scenarios. Our findings indicate that RL-based AI agents converge to a collusive state characterized by the charging of supracompetitive prices, without necessarily requiring inter-agent communication. Implementing alternative RL algorithms, altering the number of agents or simulation settings, and restricting the scope of the agents' observation space does not significantly impact the collusive market outcome behavior.
Paper Structure (11 sections, 8 equations, 5 figures)

This paper contains 11 sections, 8 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiment Design
  4. Deep Reinforcement Learning Algorithms
  5. Markov Decision Process
  6. Preprocessing and Model Architecture
  7. Experiments
  8. Scenario A: Competition
  9. Scenario B: Constrained Observation Space
  10. Discussion
  11. Conclusion

Figures (5)

  • Figure 1: Economic settings within the environment
  • Figure 2: Episode Pricing and Profit in Scenario A
  • Figure 3: Step Pricing and Profit Excerpt in Scenario A
  • Figure 4: Episode Pricing and Profit in Scenario B
  • Figure 5: Step pricing and profit excerpt of Scenario B