Bad Values but Good Behavior: Learning Highly Misspecified Bandits and MDPs

TL;DR

This work analyzes how standard decision-making algorithms can still learn near-optimal behavior under significant model misspecification. It introduces robust observation and robust parameter regions that describe when ε-greedy, LinUCB, and fitted-Q-learning maintain sublinear regret across linear bandits, contextual linear bandits, and finite-horizon MDPs, without realizability assumptions. The paper provides explicit geometric characterizations (via projection onto model subspaces) and proves sublinear regret bounds, along with detailed examples illustrating robustness regions. These results offer a theoretical explanation for the empirical success of approximate value-function methods and identify problem structures that enable robustness to misspecification.

Abstract

Parametric, feature-based reward models are employed by a variety of algorithms in decision-making settings such as bandits and Markov decision processes (MDPs). The typical assumption under which the algorithms are analysed is realizability, i.e., that the true values of actions are perfectly explained by some parametric model in the class. We are, however, interested in the situation where the true values are (significantly) misspecified with respect to the model class. For parameterized bandits, contextual bandits and MDPs, we identify structural conditions, depending on the problem instance and model class, under which basic algorithms such as $ε$-greedy, LinUCB and fitted Q-learning provably learn optimal policies under even highly misspecified models. This is in contrast to existing worst-case results for, say misspecified bandits, which show regret bounds that scale linearly with time, and shows that there can be a nontrivially large set of bandit instances that are robust to misspecification.

Figures (10)

• Figure 1: Illustration of robust regions for two function approximations
• Figure 2: An example of a MDP and a function class we designed to approximate the $Q$ value. The optimal $Q^*$ values are misspecified in the function class, yet we can learn the optimal policy using this function class.
• Figure 3: The parameter space $\mathbb{R}^2$ is partitioned into disjoint sets of the robust parameter regions corresponding to the different arms for feature matrix $\Phi = 234521$.
• Figure 4: Visualization of the robust observation regions$-\mathcal{C}_i$ for a three armed bandit problem, calculated for the feature matrix $\small{\bm{\Phi} =234521}$, along with the range space of the feature matrix $\bm{\Phi}\theta$. Note that these are $3$-dimensional plots with the robust regions $\mathcal{C}_i$ shown in shaded regions of colors blue, gray and green. These regions are subsets of $\mathbb{R}^3$ whereas the range space of the feature matrix, shown in a "plasma" color, spans $\mathbb{R}^2$.
• Figure 5: The growth of the cumulative regret for $10$ misspecified bandit instances sampled from the robust region of $\mathcal{C}_2$ under the $\varepsilon$-greedy algorithm with $\varepsilon_t = 1/\sqrt{t}$. The plot represent the average of $10$ trials. The $Y$-axis denotes the cumulative regret $\sum_{t=1}^T\mu^* - \mu_{A_t}$. The $X$-axis denotes the rounds $T$. We observe the sub-linear growth trend of the cumulative regret. For each instance the values of the $l_\infty$ misspecification error ($\rho$), the maximum sub-optimality gap ($\Delta_{\max}$) and the minimum sub-optimality gap ($\Delta_{\min}$) are also noted. It is observed that instances with higher $\Delta_{\max}$ suffer more regret at any time than instances with lower $\Delta_{\max}$ as expected from our theorem.

Theorems & Definitions (87)

• Remark 2.1: Linear Bandits
• Remark 2.2: Misspecification
• Definition 2.3: Greedy Region $\mathcal{R}$
• Remark 2.4
• Definition 2.5: Model Estimate under Sampling Distribution
• Remark 2.6
• Definition 2.7: Robust Parameter Region
• Definition 2.8: Robust Observation Region
• Theorem 2.9
• proof