Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Adjacent Similarity in Multi-Armed Bandit Tasks via Transfer of Reward Samples

NR Rahul, Vaibhav Katewa

TL;DR

The paper addresses sequential multi-task stochastic bandits with adjacently similar tasks, formalizing the per-arm mean-bound difference $|\mu_k^j-\mu_k^{j+1}|\le\epsilon_k$ and aiming to reduce cumulative regret by transferring reward samples from prior tasks. It introduces three UCB-based algorithms: NT-UCB (no transfer), Tr-UCB (known $\epsilon_k$) leveraging a bounded transfer from the preceding task, and Tr-UCB2 (unknown $\epsilon_k$) with a phased exploration to estimate similarity. The authors provide regret analyses showing no negative transfer for the transfer-based methods and demonstrate empirical gains over baselines and a naive transfer approach, particularly when task similarity is high. The results support practical transfer learning in sequential bandit settings, with implications for faster adaptation in recommender systems and online decision processes.

Abstract

We consider a sequential multi-task problem, where each task is modeled as the stochastic multi-armed bandit with K arms. We assume the bandit tasks are adjacently similar in the sense that the difference between the mean rewards of the arms for any two consecutive tasks is bounded by a parameter. We propose two algorithms (one assumes the parameter is known while the other does not) based on UCB to transfer reward samples from preceding tasks to improve the overall regret across all tasks. Our analysis shows that transferring samples reduces the regret as compared to the case of no transfer. We provide empirical results for our algorithms, which show performance improvement over the standard UCB algorithm without transfer and a naive transfer algorithm.

Exploiting Adjacent Similarity in Multi-Armed Bandit Tasks via Transfer of Reward Samples

TL;DR

The paper addresses sequential multi-task stochastic bandits with adjacently similar tasks, formalizing the per-arm mean-bound difference and aiming to reduce cumulative regret by transferring reward samples from prior tasks. It introduces three UCB-based algorithms: NT-UCB (no transfer), Tr-UCB (known ) leveraging a bounded transfer from the preceding task, and Tr-UCB2 (unknown ) with a phased exploration to estimate similarity. The authors provide regret analyses showing no negative transfer for the transfer-based methods and demonstrate empirical gains over baselines and a naive transfer approach, particularly when task similarity is high. The results support practical transfer learning in sequential bandit settings, with implications for faster adaptation in recommender systems and online decision processes.

Abstract

We consider a sequential multi-task problem, where each task is modeled as the stochastic multi-armed bandit with K arms. We assume the bandit tasks are adjacently similar in the sense that the difference between the mean rewards of the arms for any two consecutive tasks is bounded by a parameter. We propose two algorithms (one assumes the parameter is known while the other does not) based on UCB to transfer reward samples from preceding tasks to improve the overall regret across all tasks. Our analysis shows that transferring samples reduces the regret as compared to the case of no transfer. We provide empirical results for our algorithms, which show performance improvement over the standard UCB algorithm without transfer and a naive transfer algorithm.
Paper Structure (10 sections, 3 theorems, 61 equations, 4 figures, 3 algorithms)

This paper contains 10 sections, 3 theorems, 61 equations, 4 figures, 3 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Preliminaries and Problem Statement
  3. Algorithms and Regret Analysis
  4. No Transfer-UCB (NT-UCB)
  5. Transfer-UCB (Tr-UCB) with parameter $\epsilon_k$ known
  6. Transfer-UCB (Tr-UCB2) with parameter $\epsilon_k$ unknown
  7. Numerical Simulations
  8. CONCLUSIONS
  9. Proof of Theorem \ref{['theorem1_trucb']}
  10. Proof of Theorem \ref{['theorem2_trucb']}

Key Result

Lemma 1

The total pseudo-regret of NT-UCB satisfies

Figures (4)

  • Figure 1: Sequential multi-task bandit setting
  • Figure 2: Pictorial representation of Tr-UCB2
  • Figure 3: Empirical Regret Vs Total Steps of NT-UCB, Tr-UCB, Tr-UCB2 and Naive-Transfer algorithms for different values of $\epsilon_k$
  • Figure 4: Empirical Regret Vs Total Steps of NT-UCB, Tr-UCB, Tr-UCB2 and Naive-Transfer algorithms for different values of $\epsilon_k$

Theorems & Definitions (5)

  • Lemma 1
  • Remark 1
  • Theorem 1
  • Remark 2
  • Theorem 2