Multi-gear bandits, partial conservation laws, and indexability
José Niño-Mora
TL;DR
This work advances the theory and computation of index policies for dynamic projects that consume a single resource and operate across multiple gears. It introduces PCL-indexability with a verification theorem that guarantees indexability and optimality within a structured policy family, given two conditions (PCLI1 and PCLI2), and provides the downshift adaptive-greedy DS(𝔽) algorithm to compute the dynamic allocation index in AN steps. The framework yields an MP index that underpins an efficient index policy for the multi-armed multi-gear bandit problem (MAMGBP) and enables a provable bound via a Lagrangian relaxation, along with a practical downshift index policy. Extensions cover average-cost criteria, uncontrollable states, and countably infinite state spaces, highlighting the method's potential across a broad class of stochastic resource allocation problems.
Abstract
This paper considers what we propose to call multi-gear bandits, which are Markov decision processes modeling a generic dynamic and stochastic project fueled by a single resource and which admit multiple actions representing gears of operation naturally ordered by their increasing resource consumption. The optimal operation of a multi-gear bandit aims to strike a balance between project performance costs or rewards and resource usage costs, which depend on the resource price. A computationally convenient and intuitive optimal solution is available when such a model is indexable, meaning that its optimal policies are characterized by a dynamic allocation index (DAI), a function of state--action pairs representing critical resource prices. Motivated by the lack of general indexability conditions and efficient index-computing schemes, and focusing on the infinite-horizon finite-state and -action discounted case, we present a verification theorem ensuring that, if a model satisfies two proposed PCL-indexability conditions with respect to a postulated family of structured policies, then it is indexable and such policies are optimal, with its DAI being given by a marginal productivity index computed by a downshift adaptive-greedy algorithm in $A N$ steps, with $A+1$ actions and $N$ states. The DAI is further used as the basis of a new index policy for the multi-armed multi-gear bandit problem.