A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits

TL;DR

The paper delivers a comprehensive, component-driven synthesis of how large language models and multi-armed bandits intersect. By decomposing LLM systems and bandit frameworks into modular components, it clarifies how bandit strategies can enhance LLM pipelines (e.g., pre-training, alignment, prompting, RAG, and decoding) and how LLMs can enrich bandit algorithms (e.g., regret objectives, environment modeling, reward design, and action decisions). The authors propose a unified taxonomy and compile a literature index, highlighting design patterns, representative results, and open challenges. They identify two complementary research directions—Bandits for LLMs and LLMs for Bandits—and outline practical opportunities such as continual learning, adaptive prompting, multi-task and multi-objective optimization, and scalable, privacy-aware methods. Overall, the work lays a foundational framework to guide future, application-driven research at this rich intersection, with a public repository to support ongoing updates.

Abstract

Large language models (LLMs) have become powerful and widely used systems for language understanding and generation, while multi-armed bandit (MAB) algorithms provide a principled framework for adaptive decision-making under uncertainty. This survey explores the potential at the intersection of these two fields. As we know, it is the first survey to systematically review the bidirectional interaction between large language models and multi-armed bandits at the component level. We highlight the bidirectional benefits: MAB algorithms address critical LLM challenges, spanning from pre-training to retrieval-augmented generation (RAG) and personalization. Conversely, LLMs enhance MAB systems by redefining core components such as arm definition and environment modeling, thereby improving decision-making in sequential tasks. We analyze existing LLM-enhanced bandit systems and bandit-enhanced LLM systems, providing insights into their design, methodologies, and performance. Key challenges and representative findings are identified to help guide future research. An accompanying GitHub repository that indexes relevant literature is available at https://github.com/bucky1119/Awesome-LLM-Bandit-Interaction.