Variance-Optimal Arm Selection: Regret Minimization and Best Arm Identification

TL;DR

The paper tackles the problem of identifying and prioritizing the arm with the highest variance in a stochastic bandit setting, framing it through regret minimization and fixed-budget best-arm identification. It introduces UCB-VV for variance-focused regret with $O(\log n)$ guarantees and SHVV for fixed-budget BAI with an exponentially decaying error probability, both shown to be order-optimal via matching lower bounds. The authors extend the framework to sub-Gaussian rewards using a Bernstein-type concentration inequality for the sample variance and derive a path-independent concentration bound for the empirical Sharpe ratio, enabling a Sharpe-based UCB algorithm with provable regret. Extensive simulations against $\epsilon$-greedy, VTS, and KL-UCB demonstrate competitive performance, while SHVV exhibits robust BAI performance across diverse setups; a case study on call option trading with GBM-simulated stocks highlights practical applicability. Overall, the work advances variance-oriented bandit theory and provides tools for risk-aware decision making in finance-like applications, including option trading contexts.

Abstract

This paper focuses on selecting the arm with the highest variance from a set of $K$ independent arms. Specifically, we focus on two settings: (i) regret setting, that penalizes the number of pulls of suboptimal arms in terms of variance, and (ii) fixed-budget BAI setting, that evaluates the ability of an algorithm to determine the arm with the highest variance after a fixed number of pulls. We develop a novel online algorithm called \texttt{UCB-VV} for the regret setting and show that its upper bound on regret for bounded rewards evolves as $\mathcal{O}\left(\log{n}\right)$ where $n$ is the horizon. By deriving the lower bound on the regret, we show that \texttt{UCB-VV} is order optimal. For the fixed budget BAI setting, we propose the \texttt{SHVV} algorithm. We show that the upper bound of the error probability of \texttt{SHVV} evolves as $\exp\left(-\frac{n}{\log(K) H}\right)$, where $H$ represents the complexity of the problem, and this rate matches the corresponding lower bound. We extend the framework from bounded distributions to sub-Gaussian distributions using a novel concentration inequality on the sample variance. Leveraging the same, we derive a concentration inequality for the empirical Sharpe ratio (SR) for sub-Gaussian distributions, which was previously unknown in the literature. Empirical simulations show that \texttt{UCB-VV} consistently outperforms \texttt{$ε$-greedy} across different sub-optimality gaps, though it is surpassed by \texttt{VTS}, which exhibits the lowest regret, albeit lacking in theoretical guarantees. We also illustrate the superior performance of \texttt{SHVV}, for a fixed budget setting under 6 different setups against uniform sampling. Finally, we conduct a case study to empirically evaluate the performance of the \texttt{UCB-VV} and \texttt{SHVV} in call option trading on $100$ stocks generated using geometric Brownian motion (GBM).

Figures (4)

• Figure 1: Regret v/s time for UCB-VV, $\epsilon$-Greedy and VTS for (a) $\delta =0.1$ and (b) $\delta =0.5$ (c) UCB-VV v/s KL-UCB.
• Figure 2: Error probability $e_n$ of $\texttt{SHVV}$ and uniform sampling algorithm for 6 experimental setups defined.
• Figure 3: Flow Chart for call option and trade execution using SHVV and UCB-VV.
• Figure 4: Cumulative reward (profit) v/s trading period of GBM simulated $100$ stocks for the proposed strategy and UCB.

Theorems & Definitions (16)

• Definition 1
• Remark 1
• lemma 1
• Theorem 1
• Definition 2
• Definition 3
• lemma 2
• Theorem 2
• lemma 3
• lemma 4