Learning Dynamic Mechanisms in Unknown Environments: A Reinforcement Learning Approach

TL;DR

This work addresses learning a dynamic VCG mechanism in unknown, time-evolving environments by casting agent–seller interactions as an episodic linear MDP and applying reward-free online RL. The proposed Markov VCG with Linear MDs (VCG-LinMDP) uses reward-free exploration to cover the policy space and then exploitation via optimistic/pessimistic least-squares value iteration to estimate welfare-optimal policies and the VCG-like prices for agents. Theoretical results establish sublinear regret of order $\tilde{\mathcal{O}}(T^{2/3})$ for welfare, agent, and seller utilities, plus a matching minimax lower bound, thereby proving efficient learning without prior model knowledge. The framework also demonstrates approximate truthfulness, individual rationality, and efficiency in expectation, with concrete guidance on exploration duration and price construction for practical dynamic market settings. Overall, the paper provides a first provably efficient RL approach to learning dynamic, unknown-parameter mechanisms under a linear-MDP assumption, with implications for dynamic auctions, platform allocation, and public-service settings.

Abstract

Dynamic mechanism design studies how mechanism designers should allocate resources among agents in a time-varying environment. We consider the problem where the agents interact with the mechanism designer according to an unknown Markov Decision Process (MDP), where agent rewards and the mechanism designer's state evolve according to an episodic MDP with unknown reward functions and transition kernels. We focus on the online setting with linear function approximation and propose novel learning algorithms to recover the dynamic Vickrey-Clarke-Grove (VCG) mechanism over multiple rounds of interaction. A key contribution of our approach is incorporating reward-free online Reinforcement Learning (RL) to aid exploration over a rich policy space to estimate prices in the dynamic VCG mechanism. We show that the regret of our proposed method is upper bounded by $\tilde{\mathcal{O}}(T^{2/3})$ and further devise a lower bound to show that our algorithm is efficient, incurring the same $Ω(T^{2 / 3})$ regret as the lower bound, where $T$ is the total number of rounds. Our work establishes the regret guarantee for online RL in solving dynamic mechanism design problems without prior knowledge of the underlying model.

Figures (1)

• Figure 1: An illustration of the episodic MDPs $\mathcal{M}_{0},\mathcal{M}_{1}$ with the state space ${\mathcal{S}} = \{ x_0, x_1, \cdots, x_{n+2}\}$ and action space $\mathcal{A}= \{ b_j \}_{j=1}^A$. Here we fix the initial state as $x_1 = x_0$, where the agent takes the action $a \in \mathcal{A}$ and transitions into the second state $s_2\in \{x_1 , \cdots, x_{n+2}\}$. In both MDPs, we have the same transition kernel. At the first step $h=1$, the transition kernel satisfies $\mathcal{P}_{1}(x_{i}|x_{0},b_{i})=1$ for all $i\in\{1,2 , \cdots, n+1\}$ and $\mathcal{P}_{1}(x_{n+2}|x_{0},b_{i})=1$ for all $i\in\{n+2 , \cdots, A\}$. Also, $x_1, x_2, s x_{n+2} \in {\mathcal{S}}$ are the absorbing states. The reward functions for $\mathcal{M}_{0},\mathcal{M}_{1}$ are showed as in Equations \ref{['equa:lower_bound_reward_0']} and \ref{['equa:lower_bound_reward_1']}.

Theorems & Definitions (23)

• Lemma 2.1
• Remark 2.2
• Remark 3.1
• Remark 3.2
• Remark 3.3
• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Lemma C.1
• Definition D.1