Low-Rank Bandits via Tight Two-to-Infinity Singular Subspace Recovery

TL;DR

This work tackles contextual bandits with a low-rank reward structure, where each interaction reveals a single entry of the unknown $M\in\mathbb{R}^{m\times n}$ of rank $r$. It introduces a two-phase framework that first performs spectral subspace recovery of the left and right singular vectors in the two-to-infinity norm, and then solves a misspecified linear bandit in a reduced $d=r(m+n)-r^2$-dimensional space. Using tight $2\to\infty$-norm guarantees, the authors derive near-minimax results for policy evaluation and best policy identification, achieving sample complexities that scale with $(m+n)$ rather than $mn$, and provide minimax-like regret bounds for the misspecified linear-bandit reduction. The regret guarantee $R(T)=\tilde{O}\big(r^{7/4}(m+n)^{3/4}\sqrt{T}\big)$, and the policy evaluation/best-policy bounds, demonstrate substantial improvements over prior contextual low-rank approaches, especially in the homogeneous setting. The paper also introduces RS-PE, RS-BPI, and RS-RMIN algorithms that exploit the spectral phase to obtain efficient, scalable performance, with practical implications for large-scale recommendation and decision-making problems where low-rank reward structures are natural.

Abstract

We study contextual bandits with low-rank structure where, in each round, if the (context, arm) pair $(i,j)\in [m]\times [n]$ is selected, the learner observes a noisy sample of the $(i,j)$-th entry of an unknown low-rank reward matrix. Successive contexts are generated randomly in an i.i.d. manner and are revealed to the learner. For such bandits, we present efficient algorithms for policy evaluation, best policy identification and regret minimization. For policy evaluation and best policy identification, we show that our algorithms are nearly minimax optimal. For instance, the number of samples required to return an $\varepsilon$-optimal policy with probability at least $1-δ$ typically scales as ${r(m+n)\over \varepsilon^2}\log(1/δ)$. Our regret minimization algorithm enjoys minimax guarantees typically scaling as $r^{7/4}(m+n)^{3/4}\sqrt{T}$, which improves over existing algorithms. All the proposed algorithms consist of two phases: they first leverage spectral methods to estimate the left and right singular subspaces of the low-rank reward matrix. We show that these estimates enjoy tight error guarantees in the two-to-infinity norm. This in turn allows us to reformulate our problems as a misspecified linear bandit problem with dimension roughly $r(m+n)$ and misspecification controlled by the subspace recovery error, as well as to design the second phase of our algorithms efficiently.

Figures (6)

• Figure 1: Impact of data splitting ($m=n=50$, $r=2$).
• Figure 2: Impact of regularization ($m=n=50$, $r=2$)
• Figure 3: Sample size scaling of the PE error ($m=n=50$, $r=2$)
• Figure 4: Matrix size scaling of the PE error ($T=10 \ 000$, $r=1$)
• Figure 5: Matrix size scaling of the max-norm error ($T=10 \ 000$, $r=1$)

Theorems & Definitions (83)

• Definition 3.1: Homogeneity
• Theorem 4.1: Subspace recovery in $\Vert \cdot \Vert_{2 \to \infty}$
• Corollary 5.1: Misspecification error
• Theorem 6.1
• Theorem 6.2
• Theorem 6.3
• Proposition 6.4
• Theorem 7.1
• Theorem 8.1
• proof : Proof sketch of Theorem \ref{['thm:main_ESRED_context']}