Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
Marc Brooks, Gabriel Durham, Kihyuk Hong, Ambuj Tewari
TL;DR
This work addresses online personalization when actions trigger high-dimensional, generator-produced treatments by introducing GAMBITTS, a two-stage Thompson sampling framework that models both the treatment-generation process and the reward mechanism. By projecting GenAI outputs to a low-dimensional treatment embedding and leveraging online or offline treatment learning (including ensemble approaches), the method achieves regret improvements over standard bandits in GenAI-mediated interventions. Theoretical guarantees decompose uncertainty from treatment and reward, showing conditions where GAMBITTS provides stronger guarantees, especially when the embedding is well-specified and reward noise dominates. Empirical results in mHealth-like simulations demonstrate robust performance gains across various settings, highlighting the practical potential for faster, more personalized GenAI-powered interventions while noting considerations around embedding misspecification and safety. The framework opens pathways for scalable, causal-aware adaptive interventions in healthcare, education, and marketing that leverage real-time GenAI generation.
Abstract
Recent advances in generative artificial intelligence (GenAI) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems are often modeled using bandit formulations, the integration of GenAI introduces new structure into otherwise classical sequential learning problems. In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-mediated bandit-Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.