Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Learning for Combinatorial Multi-armed Bandits

Xutong Liu, Xiangxiang Dai, Jinhang Zuo, Siwei Wang, Carlee Joe-Wong, John C. S. Lui, Wei Chen

TL;DR

This work introduces Off-CMAB, a pioneering offline learning framework for combinatorial CMAB-T problems, and the Combinatorial Lower Confidence Bound (CLCB) algorithm that propagates base-arm pessimism through a combinatorial solver. It defines two data-coverage TPM conditions to quantify offline data quality and proves near-optimal suboptimality gaps that match the lower bounds up to logarithmic factors. The framework is validated across learning-to-rank, LLM caching, and influence maximization with node-level feedback, demonstrating robustness to nonlinear rewards, general feedback, and out-of-distribution actions. The results establish a principled, scalable approach for offline decision-making in large combinatorial action spaces with practical impact on recommendations, caching, and network influence tasks.

Abstract

The combinatorial multi-armed bandit (CMAB) is a fundamental sequential decision-making framework, extensively studied over the past decade. However, existing work primarily focuses on the online setting, overlooking the substantial costs of online interactions and the readily available offline datasets. To overcome these limitations, we introduce Off-CMAB, the first offline learning framework for CMAB. Central to our framework is the combinatorial lower confidence bound (CLCB) algorithm, which combines pessimistic reward estimations with combinatorial solvers. To characterize the quality of offline datasets, we propose two novel data coverage conditions and prove that, under these conditions, CLCB achieves a near-optimal suboptimality gap, matching the theoretical lower bound up to a logarithmic factor. We validate Off-CMAB through practical applications, including learning to rank, large language model (LLM) caching, and social influence maximization, showing its ability to handle nonlinear reward functions, general feedback models, and out-of-distribution action samples that excludes optimal or even feasible actions. Extensive experiments on synthetic and real-world datasets further highlight the superior performance of CLCB.

Offline Learning for Combinatorial Multi-armed Bandits

TL;DR

This work introduces Off-CMAB, a pioneering offline learning framework for combinatorial CMAB-T problems, and the Combinatorial Lower Confidence Bound (CLCB) algorithm that propagates base-arm pessimism through a combinatorial solver. It defines two data-coverage TPM conditions to quantify offline data quality and proves near-optimal suboptimality gaps that match the lower bounds up to logarithmic factors. The framework is validated across learning-to-rank, LLM caching, and influence maximization with node-level feedback, demonstrating robustness to nonlinear rewards, general feedback, and out-of-distribution actions. The results establish a principled, scalable approach for offline decision-making in large combinatorial action spaces with practical impact on recommendations, caching, and network influence tasks.

Abstract

The combinatorial multi-armed bandit (CMAB) is a fundamental sequential decision-making framework, extensively studied over the past decade. However, existing work primarily focuses on the online setting, overlooking the substantial costs of online interactions and the readily available offline datasets. To overcome these limitations, we introduce Off-CMAB, the first offline learning framework for CMAB. Central to our framework is the combinatorial lower confidence bound (CLCB) algorithm, which combines pessimistic reward estimations with combinatorial solvers. To characterize the quality of offline datasets, we propose two novel data coverage conditions and prove that, under these conditions, CLCB achieves a near-optimal suboptimality gap, matching the theoretical lower bound up to a logarithmic factor. We validate Off-CMAB through practical applications, including learning to rank, large language model (LLM) caching, and social influence maximization, showing its ability to handle nonlinear reward functions, general feedback models, and out-of-distribution action samples that excludes optimal or even feasible actions. Extensive experiments on synthetic and real-world datasets further highlight the superior performance of CLCB.

Paper Structure

This paper contains 30 sections, 67 equations, 4 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Combinatorial Multi-armed Bandits with Probabilistically Triggered Arms
  4. Offline Data Collection and Performance Metric
  5. Data Coverage Conditions: Quality of the Dataset
  6. CLCB Algorithm and Theoretical Analysis
  7. Applications of the Off-CMAB Framework
  8. Offline Learning for Cascading Bandits
  9. Offline Learning for LLM Cache
  10. Offline Learning for Influence Maximization with Extension to Node-level Feedback
  11. Experiments
  12. Conclusion and Future Directions
  13. Extended Related Works
  14. Combinatorial Multi-armed Bandits
  15. Offline Bandit and Reinforcement Learning
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Suboptimality gaps for cascading bandit application.
  • Figure 2: Comparison of different offline data generation methods.
  • Figure 3: Suboptimality gaps for LLM cache application.
  • Figure 4: Algorithms on another LLM.

Theorems & Definitions (10)

  • proof : Proof Idea
  • proof : Proof of \ref{['thm:upper_bound']}
  • proof : Proof of \ref{['thm:lower_bound']}
  • proof : Proof of \ref{['thm:app_ranking']}
  • proof
  • proof
  • proof : Proof for \ref{['thm:IM']}
  • proof
  • proof
  • proof