Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Task Diversity: Provable Representation Transfer for Sequential Multi-Task Linear Bandits

Thang Duong, Zhi Wang, Chicheng Zhang

TL;DR

The paper tackles sequential multitask linear bandits where each task parameter lies in an unknown $m$-dimensional subspace of $\mathbb{R}^d$ without assuming task diversity. It introduces the BOSS algorithm, a bi-level approach combining per-task meta-exploration and subspace-informed meta-exploitation with online subspace selection over an $\varepsilon$-cover, yielding a meta-regret of $\tilde{O}\big(Nm\sqrt{\tau} + N^{2/3}\tau^{2/3}dm^{1/3} + Nd^2 + \tau md\big)$ under ellipsoid action sets. The work demonstrates how to balance exploration and subspace learning to transfer representations online, without strong diversity assumptions, and reports empirical gains on synthetic data. This advances practical lifelong representation transfer in sequential decision-making settings with high-dimensional observations.

Abstract

We study lifelong learning in linear bandits, where a learner interacts with a sequence of linear bandit tasks whose parameters lie in an $m$-dimensional subspace of $\mathbb{R}^d$, thereby sharing a low-rank representation. Current literature typically assumes that the tasks are diverse, i.e., their parameters uniformly span the $m$-dimensional subspace. This assumption allows the low-rank representation to be learned before all tasks are revealed, which can be unrealistic in real-world applications. In this work, we present the first nontrivial result for sequential multi-task linear bandits without the task diversity assumption. We develop an algorithm that efficiently learns and transfers low-rank representations. When facing $N$ tasks, each played over $τ$ rounds, our algorithm achieves a regret guarantee of $\tilde{O}\big (Nm \sqrtτ + N^{\frac{2}{3}} τ^{\frac{2}{3}} d m^{\frac13} + Nd^2 + τm d \big)$ under the ellipsoid action set assumption. This result can significantly improve upon the baseline of $\tilde{O} \left (Nd \sqrtτ\right)$ that does not leverage the low-rank structure when the number of tasks $N$ is sufficiently large and $m \ll d$. We also demonstrate empirically on synthetic data that our algorithm outperforms baseline algorithms, which rely on the task diversity assumption.

Beyond Task Diversity: Provable Representation Transfer for Sequential Multi-Task Linear Bandits

TL;DR

The paper tackles sequential multitask linear bandits where each task parameter lies in an unknown -dimensional subspace of without assuming task diversity. It introduces the BOSS algorithm, a bi-level approach combining per-task meta-exploration and subspace-informed meta-exploitation with online subspace selection over an -cover, yielding a meta-regret of under ellipsoid action sets. The work demonstrates how to balance exploration and subspace learning to transfer representations online, without strong diversity assumptions, and reports empirical gains on synthetic data. This advances practical lifelong representation transfer in sequential decision-making settings with high-dimensional observations.

Abstract

We study lifelong learning in linear bandits, where a learner interacts with a sequence of linear bandit tasks whose parameters lie in an -dimensional subspace of , thereby sharing a low-rank representation. Current literature typically assumes that the tasks are diverse, i.e., their parameters uniformly span the -dimensional subspace. This assumption allows the low-rank representation to be learned before all tasks are revealed, which can be unrealistic in real-world applications. In this work, we present the first nontrivial result for sequential multi-task linear bandits without the task diversity assumption. We develop an algorithm that efficiently learns and transfers low-rank representations. When facing tasks, each played over rounds, our algorithm achieves a regret guarantee of under the ellipsoid action set assumption. This result can significantly improve upon the baseline of that does not leverage the low-rank structure when the number of tasks is sufficiently large and . We also demonstrate empirically on synthetic data that our algorithm outperforms baseline algorithms, which rely on the task diversity assumption.
Paper Structure (28 sections, 14 theorems, 54 equations, 3 figures, 2 tables, 4 algorithms)

This paper contains 28 sections, 14 theorems, 54 equations, 3 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Parallel representation transfer for multi-task linear bandit.
  4. Sequential representation transfer for multi-task linear bandit.
  5. Problem setup
  6. Algorithm
  7. High-level idea of our approach.
  8. The lower level.
  9. The upper level.
  10. Performance Guarantees
  11. Comparison with the individual single task baseline.
  12. Comparison with lower bounds.
  13. Adaptivity to problem parameters.
  14. Experiments
  15. Discussion and Future Work
  16. ...and 13 more sections

Key Result

Lemma 2

Fix $\tau_1$ to be a multiple of $d$. Suppose Algorithm alg:exr_procedure is run on task $n$ with the exploration length $\tau_1$. Then, there exists some constants $c_1, c_2 > 0$ (that depend on $\lambda_0, \theta_{\max}, \theta_{\min}$, and $M$) such that:

Figures (3)

  • Figure 1: Comparing the cumulative regret of BOSS and other baselines. The setting is $(N, \tau, d, m) = (4000, 500, 10, 3)$ and $\|\theta_n\|_2 \in [0.8, 1] \; \forall n \in [N]$ chosen uniformly at random from this interval. The environment only reveals a new subspace dimension at tasks 1, 2501, and 3501, so there's no task diversity assumption.
  • Figure 2: Comparing the cumulative regret of BOSS and other baselines. The setting is $(N, \tau, d, m) = (6000, 2000, 10, 3)$ and $\|\theta_n\|_2 \in [0.8, 1] \; \forall n \in [N]$ chosen uniformly at random from this interval. SeqRepL, BOSS, and BOSS-no-oracle uses the same hyperparameters $\tau_1=400, \tau_2=50$. The environment only reveals a new subspace dimension at tasks 1, 501, and 1001, and only reveals the same dimension at qin2022non's deterministic exploration schedule.
  • Figure 3: Comparing the cumulative regret of BOSS and other baselines. The setting is $(N, \tau, d, m) = (6000, 2000, 10, 3)$ and $\|\theta_n\|_2 \in [0.8, 1] \; \forall n \in [N]$. SeqRepL, BOSS, and BOSS-no-oracle uses the same hyperparameters $\tau_1=1000, \tau_2=300$. The task diversity assumption is satisfied: each $\theta_n$ is generated by a linear combinations of the columns in $B_n$ -- the subspace spanning $\theta_1, \cdots, \theta_{n-1}$. The performance of SeqRepL and BOSS is almost identical in the left figure.

Theorems & Definitions (19)

  • Lemma 2
  • Lemma 2
  • Definition 3
  • Lemma 3
  • Remark 1
  • Theorem 4
  • Definition 5
  • Definition 6
  • Lemma 7
  • Remark 2
  • ...and 9 more