Decision Making in Hybrid Environments: A Model Aggregation Approach

TL;DR

This paper addresses DEC-based characterizations of online decision making in hybrid environments where the transition dynamics are fixed but rewards may be adversarial. It develops a partition-based framework that aggregates models and policies to trade estimation complexity for decision complexity, extending both model-based and model-free learning and introducing a hybrid extension of bilinear classes for adversarial rewards. The authors show that, when the reward set is convex, the hybrid setting incurs essentially the same complexity as the stochastic regime, with only a $\log(|\Pi||\mathcal{P}|)$ estimation overhead, and they establish a $\sqrt{T}$ regret bound for linear $Q^*/V^*$ MDPs in model-free stochastic settings (improving on previous $T^{2/3}$ rates). They also provide new bounds that leverage partitioning to reduce estimation costs while preserving decision performance, contributing to a unified DEC framework with practical implications for hybrid online learning. Collectively, the results advance the understanding of decision-making under partial adversarial dynamics and offer algorithmic guidance for systems with fixed structure but evolving rewards.

Abstract

Recent work by Foster et al. (2021, 2022, 2023b) and Xu and Zeevi (2023) developed the framework of decision estimation coefficient (DEC) that characterizes the complexity of general online decision making problems and provides a general algorithm design principle. These works, however, either focus on the pure stochastic regime where the world remains fixed over time, or the pure adversarial regime where the world arbitrarily changes over time. For the hybrid regime where the dynamics of the world is fixed while the reward arbitrarily changes, they only give pessimistic bounds on the decision complexity. In this work, we propose a general extension of DEC that more precisely characterizes this case. Besides applications in special cases, our framework leads to a flexible algorithm design where the learner learns over subsets of the hypothesis set, trading estimation complexity with decision complexity, which could be of independent interest. Our work covers model-based learning and model-free learning in the hybrid regime, with a newly proposed extension of the bilinear classes (Du et al., 2021) to the adversarial-reward case. In addition, our method improves the best-known regret bounds for linear Q*/V* MDPs in the pure stochastic regime.