Policy Optimization via Adv2: Adversarial Learning on Advantage Functions
Matthieu Jonckheere, Chiara Mignacco, Gilles Stoltz
TL;DR
This paper analyzes policy optimization in episodic adversarial MDPs by formalizing a reduction to adversarial learning that can use a broad class of strategies beyond exponential weights, and by leveraging advantage functions (or $Q$-values) as inputs to these learners. It introduces Adv2, a stage-wise adversarial-learning scheme that yields regret bounds of the form $H(H+1) B_{T,A}$ when the underlying learner controls $A$-dimensional adversarial regret $B_{T,A}$. The authors present three key extensions: convergence of the last iterate under monotone-weight strategies, stronger regret notions (strongly adaptive and tracking regret) with corresponding transfers to value-function regret, and aggregation (or orchestration) of expert policies via a lifted MDP framework. They also analyze the special case of exponential weights with improved regret bounds in the episodic setting and discuss practical implications for settings with unknown transition kernels and estimated value functions, outlining directions for empirical validation of alternative policy-improvement steps. Overall, the work broadens the toolkit for adversarial MDP policy optimization and provides a roadmap for leveraging a variety of online-learning strategies while preserving theoretical guarantees, with practical implications for whether to use $Q$-values or advantage functions and how to incorporate policy aggregation.
Abstract
We revisit the reduction of learning in adversarial Markov decision processes [MDPs] to adversarial learning based on $Q$--values; this reduction has been considered in a number of recent articles as one building block to perform policy optimization. Namely, we first consider and extend this reduction in an ideal setting where an oracle provides value functions: it may involve any adversarial learning strategy (not just exponential weights) and it may be based indifferently on $Q$--values or on advantage functions. We then present two extensions: on the one hand, convergence of the last iterate for a vast class of adversarial learning strategies (again, not just exponential weights), satisfying a property called monotonicity of weights; on the other hand, stronger regret criteria for learning in MDPs, inherited from the stronger regret criteria of adversarial learning called strongly adaptive regret and tracking regret. Third, we demonstrate how adversarial learning, also referred to as aggregation of experts, relates to aggregation (orchestration) of expert policies: we obtain stronger forms of performance guarantees in this setting than existing ones, via yet another, simple reduction. Finally, we discuss the impact of the reduction of learning in adversarial MDPs to adversarial learning in the practical scenarios where transition kernels are unknown and value functions must be learned. In particular, we review the literature and note that many strategies for policy optimization feature a policy-improvement step based on exponential weights with estimated $Q$--values. Our main message is that this step may be replaced by the application of any adversarial learning strategy on estimated $Q$--values or on estimated advantage functions. We leave the empirical evaluation of these twists for future research.