Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learning

Hossein Zakerinia, Dorsa Ghobadi, Christoph H. Lampert

TL;DR

The paper tackles the puzzle of generalization in deep multi-task learning under overparameterization by introducing amortized intrinsic dimensionality (AID), a low-dimensional, shared subspace representation learned across tasks together with task-specific coefficients. It formalizes AID, demonstrates that it can be substantially smaller than single-task intrinsic dimensionality in real-world task sets, and develops encoding-based generalization bounds (including a fast-rate variant) that constrain multi-task risk using joint encoding lengths. The bounds are numerically computable via quantized encodings and codebooks, and the authors show non-vacuous guarantees on standard datasets, including transfer learning scenarios that leverage the shared representation $Q$. Collectively, the work provides both a lens to understand why deep MTL generalizes well and practical, computable guarantees that guide architecture design, representation learning, and transfer learning in multi-task settings.

Abstract

Deep learning methods are known to generalize well from training to future data, even in an overparametrized regime, where they could easily overfit. One explanation for this phenomenon is that even when their *ambient dimensionality*, (i.e. the number of parameters) is large, the models' *intrinsic dimensionality* is small; specifically, their learning takes place in a small subspace of all possible weight configurations. In this work, we confirm this phenomenon in the setting of *deep multi-task learning*. We introduce a method to parametrize multi-task network directly in the low-dimensional space, facilitated by the use of *random expansions* techniques. We then show that high-accuracy multi-task solutions can be found with much smaller intrinsic dimensionality (fewer free parameters) than what single-task learning requires. Subsequently, we show that the low-dimensional representations in combination with *weight compression* and *PAC-Bayesian* reasoning lead to the *first non-vacuous generalization bounds* for deep multi-task networks.

From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learning

TL;DR

The paper tackles the puzzle of generalization in deep multi-task learning under overparameterization by introducing amortized intrinsic dimensionality (AID), a low-dimensional, shared subspace representation learned across tasks together with task-specific coefficients. It formalizes AID, demonstrates that it can be substantially smaller than single-task intrinsic dimensionality in real-world task sets, and develops encoding-based generalization bounds (including a fast-rate variant) that constrain multi-task risk using joint encoding lengths. The bounds are numerically computable via quantized encodings and codebooks, and the authors show non-vacuous guarantees on standard datasets, including transfer learning scenarios that leverage the shared representation . Collectively, the work provides both a lens to understand why deep MTL generalizes well and practical, computable guarantees that guide architecture design, representation learning, and transfer learning in multi-task settings.

Abstract

Deep learning methods are known to generalize well from training to future data, even in an overparametrized regime, where they could easily overfit. One explanation for this phenomenon is that even when their *ambient dimensionality*, (i.e. the number of parameters) is large, the models' *intrinsic dimensionality* is small; specifically, their learning takes place in a small subspace of all possible weight configurations. In this work, we confirm this phenomenon in the setting of *deep multi-task learning*. We introduce a method to parametrize multi-task network directly in the low-dimensional space, facilitated by the use of *random expansions* techniques. We then show that high-accuracy multi-task solutions can be found with much smaller intrinsic dimensionality (fewer free parameters) than what single-task learning requires. Subsequently, we show that the low-dimensional representations in combination with *weight compression* and *PAC-Bayesian* reasoning lead to the *first non-vacuous generalization bounds* for deep multi-task networks.

Paper Structure

This paper contains 24 sections, 9 theorems, 24 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Intrinsic dimensionality of neural networks
  4. Amortized intrinsic dimensionality for deep multi-task learning
  5. Determining the AID for real-world tasks
  6. Non-vacuous generalization guarantees for multi-task learning
  7. Background and related work
  8. Compression-based guarantees for multi-task learning
  9. Discussion.
  10. Computing the bounds numerically
  11. Encoding all tasks together
  12. Application to transfer learning
  13. Related work
  14. Conclusion
  15. Proofs
  16. ...and 9 more sections

Key Result

Theorem 1

Let $E$ be a model encoding scheme with length function $l_E:\bigcup_{n=1}^{\infty} \mathcal{F}^n\to\mathbb{N}$. Then, for any $\delta>0$, the following inequality holds with probability at least $1-\delta$ (over the random training data of size $m$): for all $f\in\mathcal{F}$

Figures (1)

  • Figure 1: Amortized intrinsic dimensionality, $\text{AID}_{90}$, for multi-task learning on the MNIST PL (left) and MNIST SP (right) datasets ($n=10,20,\dots,100$ tasks, $m=600$ samples per task). The more tasks are available, the fewer learnable parameters per task are required.

Theorems & Definitions (16)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 1: shalev2014understanding, Theorem 7.7
  • Definition 4
  • Theorem 2
  • Theorem 3
  • Corollary 4
  • Theorem 4
  • proof
  • ...and 6 more