Survey: Multi-Armed Bandits Meet Large Language Models

TL;DR

The paper addresses integrating bandit optimization with LLMs to improve data efficiency, prompting, and adaptive reasoning. It surveys how bandits can enhance LLM training, prompting, and evaluation, while LLMs can enrich bandit decision-making with contextual understanding and natural-language feedback. Key contributions include a structured taxonomy of techniques (data selection, hyperparameter tuning, CoT with dueling bandits, contextual prompting) and a roadmap for future research. The work highlights the potential for more efficient, adaptive, and user-centric AI systems across domains such as recommendations, dialogue, healthcare, and autonomous systems.

Abstract

Bandit algorithms and Large Language Models (LLMs) have emerged as powerful tools in artificial intelligence, each addressing distinct yet complementary challenges in decision-making and natural language processing. This survey explores the synergistic potential between these two fields, highlighting how bandit algorithms can enhance the performance of LLMs and how LLMs, in turn, can provide novel insights for improving bandit-based decision-making. We first examine the role of bandit algorithms in optimizing LLM fine-tuning, prompt engineering, and adaptive response generation, focusing on their ability to balance exploration and exploitation in large-scale learning tasks. Subsequently, we explore how LLMs can augment bandit algorithms through advanced contextual understanding, dynamic adaptation, and improved policy selection using natural language reasoning. By providing a comprehensive review of existing research and identifying key challenges and opportunities, this survey aims to bridge the gap between bandit algorithms and LLMs, paving the way for innovative applications and interdisciplinary research in AI.