Multiagent Reinforcement Learning for Liquidity Games

TL;DR

The paper tackles how to achieve high market liquidity as an emergent property of decentralized, self-interested agents in bilateral markets, where explicit coordination is illegal. It introduces a unified framework that couples Liquidity Games with Rational Swarms by using difference rewards within a Markov team-game to align individual learning with a global liquidity objective $G$. The authors show that difference rewards enable independent agents to improve both their own trading outcomes and the overall liquidity across MinFill and exact matching regimes, without centralized coordination, supported by a tabular Q-learning study and baseline comparisons. The findings suggest a scalable approach to decentralized market design, with practical implications for liquidity provision incentives in government-bond and similar bilateral markets. This work contributes a concrete methodology to connect micro-level agent behavior with macro-level market efficiency, offering a path toward robust, coordination-free market design.

Abstract

Making use of swarm methods in financial market modeling of liquidity, and techniques from financial analysis in swarm analysis, holds the potential to advance both research areas. In swarm research, the use of game theory methods holds the promise of explaining observed phenomena of collective utility adherence with rational self-interested swarm participants. In financial markets, a better understanding of how independent financial agents may self-organize for the betterment and stability of the marketplace would be a boon for market design researchers. This paper unifies Liquidity Games, where trader payoffs depend on aggregate liquidity within a trade, with Rational Swarms, where decentralized agents use difference rewards to align self-interested learning with global objectives. We offer a theoretical frameworks where we define a swarm of traders whose collective objective is market liquidity provision while maintaining agent independence. Using difference rewards within a Markov team games framework, we show that individual liquidity-maximizing behaviors contribute to overall market liquidity without requiring coordination or collusion. This Financial Swarm model provides a framework for modeling rational, independent agents where they achieve both individual profitability and collective market efficiency in bilateral asset markets.

Figures (6)

• Figure 1: Schematic of the Liquidity Swarm model. Agents $i$ and $j$ interact via bilateral trades. Actions affect global liquidity, which feeds into local reward signals via difference rewards.
• Figure 2: Learning agents surpassing baselines.
• Figure 3: Cumulative liquidity through episodes for MinFill rules
• Figure 4: Hit ratios are high for learning algorithms, shown here for the exact fill rule (note: by construction, for MinFill they are 100%). Note: Learning ($exact_diff$) and learning ($exact_local$) are almost identical results and graphs overwrite each other
• Figure 5: Clearing rates clearly converge early with learning, and difference functions in the exact rule being identical to local learning.

Theorems & Definitions (1)

• Definition 1: Liquidity Game