Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Local Regularizers Are Not Transductive Learners

Sky Jafar, Julian Asilis, Shaddin Dughmi

TL;DR

This work addresses whether local regularization (local SRM) can learn all learnable multiclass problems. It constructs a learnable hypothesis class $ abla H_{otp}$ based on secret-sharing (a two-output encoding related to the one-time pad) and shows that, although learnable, it cannot be transductively learned by any local regularizer, via a coupling argument that forces a cyclical preference among hypotheses. The result hinges on the class being generalized binary with distinct label sets (GBDLS) and employs a careful probabilistic construction to lower-bound transductive error to at least $1/4$. Extending the lower bound to the PAC model remains open, with three main technical challenges identified; nonetheless, the transductive separation already highlights limitations of local regularization as a universal template for multiclass learning and motivates a potential PAC/transductive separation. The paper thus advances our understanding of the expressive limits of local regularizers and connects learning theory with cryptographic ideas, while leaving open the precise PAC implications.

Abstract

We partly resolve an open question raised by Asilis et al. (COLT 2024): whether the algorithmic template of local regularization -- an intriguing generalization of explicit regularization, a.k.a. structural risk minimization -- suffices to learn all learnable multiclass problems. Specifically, we provide a negative answer to this question in the transductive model of learning. We exhibit a multiclass classification problem which is learnable in both the transductive and PAC models, yet cannot be learned transductively by any local regularizer. The corresponding hypothesis class, and our proof, are based on principles from cryptographic secret sharing. We outline challenges in extending our negative result to the PAC model, leaving open the tantalizing possibility of a PAC/transductive separation with respect to local regularization.

Local Regularizers Are Not Transductive Learners

TL;DR

This work addresses whether local regularization (local SRM) can learn all learnable multiclass problems. It constructs a learnable hypothesis class based on secret-sharing (a two-output encoding related to the one-time pad) and shows that, although learnable, it cannot be transductively learned by any local regularizer, via a coupling argument that forces a cyclical preference among hypotheses. The result hinges on the class being generalized binary with distinct label sets (GBDLS) and employs a careful probabilistic construction to lower-bound transductive error to at least . Extending the lower bound to the PAC model remains open, with three main technical challenges identified; nonetheless, the transductive separation already highlights limitations of local regularization as a universal template for multiclass learning and motivates a potential PAC/transductive separation. The paper thus advances our understanding of the expressive limits of local regularizers and connects learning theory with cryptographic ideas, while leaving open the precise PAC implications.

Abstract

We partly resolve an open question raised by Asilis et al. (COLT 2024): whether the algorithmic template of local regularization -- an intriguing generalization of explicit regularization, a.k.a. structural risk minimization -- suffices to learn all learnable multiclass problems. Specifically, we provide a negative answer to this question in the transductive model of learning. We exhibit a multiclass classification problem which is learnable in both the transductive and PAC models, yet cannot be learned transductively by any local regularizer. The corresponding hypothesis class, and our proof, are based on principles from cryptographic secret sharing. We outline challenges in extending our negative result to the PAC model, leaving open the tantalizing possibility of a PAC/transductive separation with respect to local regularization.

Paper Structure

This paper contains 10 sections, 4 theorems, 11 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. Learning Theory
  5. Local Regularization
  6. Secret Sharing
  7. GBDLS Classes
  8. Insufficiency of Local Regularization in the Transductive Model
  9. Challenges in Extending to the PAC Model
  10. Conclusion

Key Result

Proposition 3.5

Let $\mathcal{H} \subseteq \mathcal{Y}^\mathcal{X}$ be a GBDLS hypothesis class. Then $\mathcal{H}$ is learnable, in both the PAC and transductive models.

Figures (1)

  • Figure 1: Depiction of the learning problems $\{(S_i,h^*_i)\}_{i \in [4]}$ in \ref{['Theorem:local-regularization-fails']}, with training sets $S_i$ circumscribed in blue. Each row corresponds to one of the learning problems $(S_i,h^*_i)$. Crucially, success of a local regularizer $\psi$ on each learning problem imposes the ordering relations on the right-hand side, which collectively produce a cycle.

Theorems & Definitions (23)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5: asilis-open-problem
  • Remark 2.6
  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Definition 3.4
  • ...and 13 more