Contractual Reinforcement Learning: Pulling Arms with Invisible Hands
Jibang Wu, Siyu Chen, Mengdi Wang, Huazheng Wang, Haifeng Xu
TL;DR
This work introduces contractual reinforcement learning (PRL) in a principal–agent framework (PAMDP) to address incentive misalignment in online learning. It develops a planning approach via Bellman-based least-payment equations that yield a polynomial-time method for the principal to compute optimal contracts against a rational, far-sighted agent, and it designs no-regret learning algorithms that decouple contract design from policy optimization to achieve sublinear regret. For the contractual bandit case (H=1), the authors provide a generic, robust learning scheme with regret $\tilde{O}(\sqrt{T})$ under mild inducibility conditions, and they extend to the general contractive RL setting with $\tilde{O}(\sqrt{T})$ or $\tilde{O}(T^{2/3})$ regret depending on problem structure and regularity assumptions. The results illuminate the interplay between statistical estimation and computational search in contract design, and have practical implications for AI alignment and incentive-aware design in platforms where content and data collection are steered by self-interested agents.
Abstract
The agency problem emerges in today's large scale machine learning tasks, where the learners are unable to direct content creation or enforce data collection. In this work, we propose a theoretical framework for aligning economic interests of different stakeholders in the online learning problems through contract design. The problem, termed \emph{contractual reinforcement learning}, naturally arises from the classic model of Markov decision processes, where a learning principal seeks to optimally influence the agent's action policy for their common interests through a set of payment rules contingent on the realization of next state. For the planning problem, we design an efficient dynamic programming algorithm to determine the optimal contracts against the far-sighted agent. For the learning problem, we introduce a generic design of no-regret learning algorithms to untangle the challenges from robust design of contracts to the balance of exploration and exploitation, reducing the complexity analysis to the construction of efficient search algorithms. For several natural classes of problems, we design tailored search algorithms that provably achieve $\tilde{O}(\sqrt{T})$ regret. We also present an algorithm with $\tilde{O}(T^{2/3})$ for the general problem that improves the existing analysis in online contract design with mild technical assumptions.