Multi-armed Bandits with Missing Outcome

TL;DR

This paper introduces algorithms that account for missingness under both missing at random (MAR) and missing not at random (MNAR) models and demonstrates the drastic improvements in decision-making by accounting for missingness in these settings.

Abstract

While significant progress has been made in designing algorithms that minimize regret in online decision-making, real-world scenarios often introduce additional complexities, perhaps the most challenging of which is missing outcomes. Overlooking this aspect or simply assuming random missingness invariably leads to biased estimates of the rewards and may result in linear regret. Despite the practical relevance of this challenge, no rigorous methodology currently exists for systematically handling missingness, especially when the missingness mechanism is not random. In this paper, we address this gap in the context of multi-armed bandits (MAB) with missing outcomes by analyzing the impact of different missingness mechanisms on achievable regret bounds. We introduce algorithms that account for missingness under both missing at random (MAR) and missing not at random (MNAR) models. Through both analytical and simulation studies, we demonstrate the drastic improvements in decision-making by accounting for missingness in these settings.

Figures (5)

• Figure 1: Graphical representations of the missing data mechanisms considered in this paper.
• Figure 2: Special case of MAR.
• Figure 3: Graphical representations of the missing data mechanisms with missing outcome and mediator.
• Figure 4: Results corresponding to MCAR, MAR, and MNAR settings. The shaded regions represent the error bars, showing one standard deviation across multiple runs of the simulations.
• Figure 5: Complementary evaluation results for our proposed algorithms.

Theorems & Definitions (27)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Corollary 1
• Theorem 6
• Theorem 7
• Remark 1
• Theorem 7