Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Matrix Sketching in Bandits: Current Pitfalls and New Framework

Dongxie Wen, Hanyan Yin, Xiao Zhang, Zhewei Wei

TL;DR

This paper revisits the regret analysis and algorithm design concerning approximating the covariance matrix using matrix sketching in linear bandits and proposes Dyadic Block Sketching, an innovative streaming matrix sketching approach that adaptively manages sketch size to constrain global spectral loss.

Abstract

The utilization of sketching techniques has progressively emerged as a pivotal method for enhancing the efficiency of online learning. In linear bandit settings, current sketch-based approaches leverage matrix sketching to reduce the per-round time complexity from \(Ω\left(d^2\right)\) to \(O(d)\), where \(d\) is the input dimension. Despite this improved efficiency, these approaches encounter critical pitfalls: if the spectral tail of the covariance matrix does not decrease rapidly, it can lead to linear regret. In this paper, we revisit the regret analysis and algorithm design concerning approximating the covariance matrix using matrix sketching in linear bandits. We illustrate how inappropriate sketch sizes can result in unbounded spectral loss, thereby causing linear regret. To prevent this issue, we propose Dyadic Block Sketching, an innovative streaming matrix sketching approach that adaptively manages sketch size to constrain global spectral loss. This approach effectively tracks the best rank-\( k \) approximation in an online manner, ensuring efficiency when the geometry of the covariance matrix is favorable. Then, we apply the proposed Dyadic Block Sketching to linear bandits and demonstrate that the resulting bandit algorithm can achieve sublinear regret without prior knowledge of the covariance matrix, even under the worst case. Our method is a general framework for efficient sketch-based linear bandits, applicable to all existing sketch-based approaches, and offers improved regret bounds accordingly. Additionally, we conduct comprehensive empirical studies using both synthetic and real-world data to validate the accuracy of our theoretical findings and to highlight the effectiveness of our algorithm.

Matrix Sketching in Bandits: Current Pitfalls and New Framework

TL;DR

This paper revisits the regret analysis and algorithm design concerning approximating the covariance matrix using matrix sketching in linear bandits and proposes Dyadic Block Sketching, an innovative streaming matrix sketching approach that adaptively manages sketch size to constrain global spectral loss.

Abstract

The utilization of sketching techniques has progressively emerged as a pivotal method for enhancing the efficiency of online learning. In linear bandit settings, current sketch-based approaches leverage matrix sketching to reduce the per-round time complexity from \(Ω\left(d^2\right)\) to \(O(d)\), where is the input dimension. Despite this improved efficiency, these approaches encounter critical pitfalls: if the spectral tail of the covariance matrix does not decrease rapidly, it can lead to linear regret. In this paper, we revisit the regret analysis and algorithm design concerning approximating the covariance matrix using matrix sketching in linear bandits. We illustrate how inappropriate sketch sizes can result in unbounded spectral loss, thereby causing linear regret. To prevent this issue, we propose Dyadic Block Sketching, an innovative streaming matrix sketching approach that adaptively manages sketch size to constrain global spectral loss. This approach effectively tracks the best rank- approximation in an online manner, ensuring efficiency when the geometry of the covariance matrix is favorable. Then, we apply the proposed Dyadic Block Sketching to linear bandits and demonstrate that the resulting bandit algorithm can achieve sublinear regret without prior knowledge of the covariance matrix, even under the worst case. Our method is a general framework for efficient sketch-based linear bandits, applicable to all existing sketch-based approaches, and offers improved regret bounds accordingly. Additionally, we conduct comprehensive empirical studies using both synthetic and real-world data to validate the accuracy of our theoretical findings and to highlight the effectiveness of our algorithm.

Paper Structure

This paper contains 34 sections, 14 theorems, 72 equations, 5 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Organization.
  4. Revisiting Matrix Sketching in Linear Bandits
  5. Notations.
  6. Linear Bandits through Matrix Sketching
  7. The Motivation of Revisiting
  8. Dyadic Block Sketching for Constrained Global Error Bound
  9. Decomposability
  10. Algorithm Descriptions
  11. High-Level Ideas.
  12. Algorithm.
  13. Analysis
  14. Dyadic Block Sketching for FD.
  15. Linear Bandits through Dyadic Block Sketching
  16. ...and 19 more sections

Key Result

Theorem 1

Suppose the chosen arm $\bm{x}_t\in\mathbb{R}^d$ at round $t$ is a random vector drawn iid from any distribution over $r\le d$ orthonormal vectors $\bm{A}$. For any sketch size $l\le r$, the bound on the expected regret of linear bandits using FD is $\Omega\left(T^2\right)$.

Figures (5)

  • Figure 1: Regret of SOFUL, OFUL, and our method on synthetic data (details in section \ref{['sec: Online Regression in Synthetic Data']}) The regret of SOFUL is nearly linear when sketch size $l=300$.
  • Figure 2: An illustration for Dyadic Block Sketching. For inactive Block $i \in [B-1]$, the matrix sketch covers the data from $t_{i-1}$ to $t_i$. For the active Block $B$, matrix sketching updates are performed on the new rows. We then merge the multi-scale matrix sketches to approximate the entire stream.
  • Figure 3: Comparison among FD and our DBS-FD w.r.t. the error and its upper bound
  • Figure 4: (a): Cumulative regret of the compared algorithms, the proposed DBSLinUCB using RFD on a synthetic dataset; (b), (c): Cumulative regret and total running time of the compared algorithms, the proposed DBSLinUCB using FD on MNIST
  • Figure : FD sketch

Theorems & Definitions (25)

  • Theorem 1
  • Lemma 1: Decomposability
  • proof
  • Theorem 2
  • Corollary 1
  • Remark 1: Efficient Implementation
  • Remark 2: Worst-Case Analysis
  • Theorem 3
  • Remark 3
  • Theorem 4
  • ...and 15 more