Fixed-Budget Change Point Identification in Piecewise Constant Bandits

TL;DR

The paper tackles locating a single abrupt change point in a piecewise-constant reward function under a fixed sampling budget in a continuous action space. It develops two non-asymptotic algorithms, Sequential Halving with Backtracking (SHB) for large budgets and Sequential Halving (SH) for small budgets, and introduces a regime-adaptive SHA that unifies their strengths. The authors derive matching upper and lower bounds (minimax up to constants) that reveal a fundamental separation in complexity between large- and small-budget regimes and prove SHA is near-optimal across both regimes. Empirical results in simulated environments corroborate the theory, showing SHA's robustness and competitiveness against existing approaches, while highlighting the practical benefits of non-asymptotic guarantees in fixed-budget change-point identification.

Abstract

We study the piecewise constant bandit problem where the expected reward is a piecewise constant function with one change point (discontinuity) across the action space $[0,1]$ and the learner's aim is to locate the change point. Under the assumption of a fixed exploration budget, we provide the first non-asymptotic analysis of policies designed to locate abrupt changes in the mean reward function under bandit feedback. We study the problem under a large and small budget regime, and for both settings establish lower bounds on the error probability and provide algorithms with near matching upper bounds. Interestingly, our results show a separation in the complexity of the two regimes. We then propose a regime adaptive algorithm which is near optimal for both small and large budgets simultaneously. We complement our theoretical analysis with experimental results in simulated environments to support our findings.

Figures (7)

• Figure 1: Example of a piecewise constant mean reward function, $f$, across action space $[0,1]$ with change point $x^*$, change in mean of $\Delta$ from $\mu_1$ to $\mu_2$ and 10 arbitrarily chosen noisy observations in red.
• Figure 2: Example illustration of action space $\mathcal{A}=[0,1]$ in (Top) phase $j$ with sampling points $\mathcal{A}^j = \{0,a_1^j,a_2^j,a_3^j,1\}$ and (Bottom) phase $j+1$ with sampling points $\mathcal{A}^{j+1} = \{0,a_1^{j+1},a_2^{j+1},a_3^{j+1},1\}$, where the shaded regions have been eliminated. In this example, $E_{R,j}$ held and region $[a_1^j,a_2^j)$ was eliminated in phase $j$.
• Figure 3: Proportion of final estimates more than $\eta$ away from $x^*$ against the inputted budget, $T$, with Gaussian rewards, $\Delta=2$ and 90% CIs. (a) compares the SH, SHB, and SHA by running each algorithm 1000 times with different budgets. (b,c) both compare SH, SRR, and ACPD. We run SH and SRR 500 times each at different budgets, while the anytime ACPD algorithm is run a total of 500 times for $T=60$ and we plot the evolution of ACPD's failure probability.
• Figure 4: Proportion of final estimates more than $\eta$ away from $x^*$ against the inputted budget, $T$, with Gaussian rewards, $\Delta=2$ and 90% CIs. SH, SHB, and SHA were each run 1000 times with different budgets.
• Figure 5: Proportion of final estimates more than $\eta$ away from $x^*$ against the inputted budget, $T$, with Gaussian rewards, $\Delta=2$ and 90% CIs. SH, SHB, and SHA were each run 1000 times with different budgets.

Theorems & Definitions (38)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• Theorem 4
• Theorem 5
• proof
• Lemma 1
• Lemma 2