Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Infinite-Dimensional Regression: Learning Linear Operators

Vinod Raman, Unique Subedi, Ambuj Tewari

TL;DR

The paper studies online learnability of linear operators between infinite-dimensional Hilbert spaces under squared loss in an adversarial setting. It shows uniformly bounded $p$-Schatten operator classes $S_p$ are online learnable with sublinear regret $O\big(T^{\max\{1/2,1-1/p\}}\big)$, and establishes matching lower bounds $\Omega\big(T^{1-1/p}\big)$ for $p\ge 2$, while the operator-norm-bounded class is not online learnable; it also proves a separation where online learnability holds but sequential uniform convergence fails for certain operator classes, with analogous batch-setup hardness results. The analysis hinges on sequential Rademacher complexity bounds for the $p$-Schatten loss class and constructive proofs using rank-1 operator sums, suggesting efficient algorithms via online mirror descent with $\ell_p$-type regularization. Open questions include determining optimal rates for $p\in[1,2)$, designing scalable algorithms for Schatten classes, and extending the framework to general or non-linear operators, as well as clarifying the relationship between uniform convergence and learnability beyond finite-dimensional settings.

Abstract

We consider the problem of learning linear operators under squared loss between two infinite-dimensional Hilbert spaces in the online setting. We show that the class of linear operators with uniformly bounded $p$-Schatten norm is online learnable for any $p \in [1, \infty)$. On the other hand, we prove an impossibility result by showing that the class of uniformly bounded linear operators with respect to the operator norm is \textit{not} online learnable. Moreover, we show a separation between sequential uniform convergence and online learnability by identifying a class of bounded linear operators that is online learnable but uniform convergence does not hold. Finally, we prove that the impossibility result and the separation between uniform convergence and learnability also hold in the batch setting.

Online Infinite-Dimensional Regression: Learning Linear Operators

TL;DR

The paper studies online learnability of linear operators between infinite-dimensional Hilbert spaces under squared loss in an adversarial setting. It shows uniformly bounded -Schatten operator classes are online learnable with sublinear regret , and establishes matching lower bounds for , while the operator-norm-bounded class is not online learnable; it also proves a separation where online learnability holds but sequential uniform convergence fails for certain operator classes, with analogous batch-setup hardness results. The analysis hinges on sequential Rademacher complexity bounds for the -Schatten loss class and constructive proofs using rank-1 operator sums, suggesting efficient algorithms via online mirror descent with -type regularization. Open questions include determining optimal rates for , designing scalable algorithms for Schatten classes, and extending the framework to general or non-linear operators, as well as clarifying the relationship between uniform convergence and learnability beyond finite-dimensional settings.

Abstract

We consider the problem of learning linear operators under squared loss between two infinite-dimensional Hilbert spaces in the online setting. We show that the class of linear operators with uniformly bounded -Schatten norm is online learnable for any . On the other hand, we prove an impossibility result by showing that the class of uniformly bounded linear operators with respect to the operator norm is \textit{not} online learnable. Moreover, we show a separation between sequential uniform convergence and online learnability by identifying a class of bounded linear operators that is online learnable but uniform convergence does not hold. Finally, we prove that the impossibility result and the separation between uniform convergence and learnability also hold in the batch setting.
Paper Structure (18 sections, 6 theorems, 58 equations)

This paper contains 18 sections, 6 theorems, 58 equations.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Hilbert Space Basics
  5. Online Learning
  6. $p$-Schatten Operators are Online Learnable
  7. Examples of $p$-Schatten class
  8. Lower Bounds and Hardness Results
  9. Lower Bounds in the Batch Setting
  10. Online Learnability without Sequential Uniform Convergence
  11. Batch Learnability without Uniform Convergence
  12. Discussion and Open Questions
  13. Upperbound Proofs for Online Setting
  14. Proof of Lemma \ref{['radsum']}
  15. Proof of Theorem \ref{['Sp']}
  16. ...and 3 more sections

Key Result

Theorem 4

Fix $c> 0$. Let $\mathcal{X} = \{v \in \mathcal{V} \mid \left\lVert v\right\rVert \leq 1\}$ denote the instance space, $\mathcal{Y} = \{w \in \mathcal{W} \mid \left\lVert w\right\rVert \leq c\}$ denote the target space, and $\mathcal{F}_p = \{f \in S_{p}(\mathcal{V}, \mathcal{W}) \, \mid \, \left\lV

Theorems & Definitions (10)

  • Definition 1: Online Linear Operator Learnability
  • Definition 2: Sequential Uniform Convergence
  • Definition 3: Sequential Rademacher Complexity
  • Theorem 4: Uniformly Bounded Subsets of $S_p(\mathcal{V}, \mathcal{W})$ are Online Learnable
  • Lemma 5: Rademacher Sums of Rank-1 Operators
  • Theorem 6: Lower Bounds for Uniformly Bounded Subsets of $S_p(\mathcal{V}, \mathcal{W})$
  • Definition 7: Batch Learnability
  • Theorem 8: Batch Lower Bounds for Uniformly Bounded Subsets of $S_p(\mathcal{V}, \mathcal{W})$
  • Theorem 9: Sequential Uniform Convergence $\not \equiv$ Online Learnability
  • Lemma 10