Regret Minimization via Saddle Point Optimization

TL;DR

This work derives an anytime variant of the Estimation-To-Decisions (E2D) algorithm that optimizes the exploration-exploitation trade-off online instead of via the analysis, and leads to a practical algorithm for finite model classes and linear feedback models.

Abstract

A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs. In the corresponding saddle-point game, the min-player optimizes the sampling distribution against an adversarial max-player that chooses confusing models leading to large regret. The most recent instantiation of this idea is the decision-estimation coefficient (DEC), which was shown to provide nearly tight lower and upper bounds on the worst-case expected regret in structured bandits and reinforcement learning. By re-parametrizing the offset DEC with the confidence radius and solving the corresponding min-max program, we derive an anytime variant of the Estimation-To-Decisions (E2D) algorithm. Importantly, the algorithm optimizes the exploration-exploitation trade-off online instead of via the analysis. Our formulation leads to a practical algorithm for finite model classes and linear feedback models. We further point out connections to the information ratio, decoupling coefficient and PAC-DEC, and numerically evaluate the performance of E2D on simple examples.

Theorems & Definitions (11)

• Example 2.1: Linear Bandits, abe1999associative
• Example 2.2: Linear Bandits with Side-Observations
• Theorem 1
• Corollary 1
• proof : Proof of \ref{['thm:worst-case']}
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5