Market-based Architectures in RL and Beyond
Abhimanyu Pallavi Sudhir, Long Tran-Thanh
TL;DR
The paper addresses AI challenges related to search efficiency, dynamic scalability, and complete feedback for alignment by proposing market-based reinforcement-learning architectures. It introduces two frameworks, a deep market with a single state good and a wide market that partitions the state into goods $ \, \mathcal{S} = \mathcal{S}_1 \oplus \dots \oplus \mathcal{S}_n \, $, enabling specialized, parallel sub-agents, and demonstrates how neural networks can emerge as a special case of market dynamics through equilibrium-based valuations. Key contributions include formal definitions of both market models, showing a neural-network equivalence, discussing pricing via market equilibria and backpropagation through the market graph, and outlining applications to search, scalable reasoning, complete feedback, and LLMs. The work outlines practical challenges (e.g., agent enumeration and memory) and a roadmap for theory (convergence, welfare analyses) and practice (Python library, LLM-driven markets) with potential for broad impact in AI systems and alignment.
Abstract
Market-based agents refer to reinforcement learning agents which determine their actions based on an internal market of sub-agents. We introduce a new type of market-based algorithm where the state itself is factored into several axes called ``goods'', which allows for greater specialization and parallelism than existing market-based RL algorithms. Furthermore, we argue that market-based algorithms have the potential to address many current challenges in AI, such as search, dynamic scaling and complete feedback, and demonstrate that they may be seen to generalize neural networks; finally, we list some novel ways that market algorithms may be applied in conjunction with Large Language Models for immediate practical applicability.