Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memorizing Long-tail Data Can Help Generalization Through Composition

Mo Zhou, Haoyang Ma, Rong Ge

TL;DR

The paper investigates how memorization interacts with compositional generalization under long-tail feature distributions, showing memorization can aid generalization when combined with simple composition. It provides a linear-theory analysis proving that memorization enables correct predictions on unseen feature-combinations and gives out-of-distribution guarantees under power-law feature frequencies, with rates $\lesssim s(\frac{\ln^2 d}{n s})^{1-1/\alpha}$ for $\alpha>0$. Complementary experiments on linear models and simple neural-net architectures demonstrate that composition capability and the benefits of memorization depend on architecture, illustrating practical implications for designing models that generalize via composition. Overall, the work highlights memorization as a potential resource for compositional generalization in long-tail regimes, while acknowledging the need for broader theory and larger-scale nonlinear analyses.

Abstract

Deep learning has led researchers to rethink the relationship between memorization and generalization. In many settings, memorization does not hurt generalization due to implicit regularization and may help by memorizing long-tailed examples. In this paper, we consider the synergy between memorization and simple composition -- the ability to make correct prediction on a combination of long-tailed features. Theoretically, we show that for a linear setting, memorization together with composition can help the model make correct predictions on rare test examples that require a combination of long-tailed features, even if such combinations were never observed in the training data. Experiments on neural network architecture on simple data show that the theoretical insight extends beyond the linear setting, and we further observe that the composition capability of the model depends on its architecture.

Memorizing Long-tail Data Can Help Generalization Through Composition

TL;DR

The paper investigates how memorization interacts with compositional generalization under long-tail feature distributions, showing memorization can aid generalization when combined with simple composition. It provides a linear-theory analysis proving that memorization enables correct predictions on unseen feature-combinations and gives out-of-distribution guarantees under power-law feature frequencies, with rates for . Complementary experiments on linear models and simple neural-net architectures demonstrate that composition capability and the benefits of memorization depend on architecture, illustrating practical implications for designing models that generalize via composition. Overall, the work highlights memorization as a potential resource for compositional generalization in long-tail regimes, while acknowledging the need for broader theory and larger-scale nonlinear analyses.

Abstract

Deep learning has led researchers to rethink the relationship between memorization and generalization. In many settings, memorization does not hurt generalization due to implicit regularization and may help by memorizing long-tailed examples. In this paper, we consider the synergy between memorization and simple composition -- the ability to make correct prediction on a combination of long-tailed features. Theoretically, we show that for a linear setting, memorization together with composition can help the model make correct predictions on rare test examples that require a combination of long-tailed features, even if such combinations were never observed in the training data. Experiments on neural network architecture on simple data show that the theoretical insight extends beyond the linear setting, and we further observe that the composition capability of the model depends on its architecture.
Paper Structure (58 sections, 31 theorems, 80 equations, 4 figures, 2 tables)

This paper contains 58 sections, 31 theorems, 80 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Our results
  3. Related Works
  4. Memorization and long‑tail data
  5. Memorization in learning
  6. Compositional Generalization
  7. Preliminaries
  8. Notation
  9. Target Model
  10. Data
  11. Choice of parameters
  12. Algorithm and loss
  13. Performance Measurements: In- and Out-of-Distribution Test Loss
  14. Memorization helps out-of-distribution generalization
  15. Noiseless case
  16. ...and 43 more sections

Key Result

Theorem 1.1

Let $\widetilde{{\mathcal{D}}}$ be a test distribution that differs from the training distribution ${\mathcal{D}}$ only in that each sample from $\widetilde{{\mathcal{D}}}$ contains a composition of $t$ long-tail features, each of which appeared exactly once in the training data. Then, the test loss

Figures (4)

  • Figure 1: (a)(b) Average Error on Common and Long-tail Features on Linear Model; (c)(d) In- and Out-of-Distribution Test Loss on Linear Model
  • Figure 2: Test Performance Comparison Across Sum, Linear, 2-layer and Cross-channel Models
  • Figure 3: Test loss as a function of the number of training occurrences of class 9.
  • Figure 4: (Left) ResNet+Linear layer with New Target Function: Test loss as a function of the number of training occurrences of class 9; (Right) ResNet+2-layer network with New Target Function: Test loss as a function of the number of training occurrences of class 9.

Theorems & Definitions (46)

  • Theorem 1.1: Informal
  • Claim 2.1
  • Theorem 3.1: Test loss
  • Theorem 3.2: OOD performance
  • Theorem 3.3: Test loss
  • Theorem 3.4: OOD performance
  • Lemma 4.1: Structure of ${\bm{X}}$, informal of Lemma \ref{['lem: structure of X']}
  • Lemma 4.2: Recovery of $\beta^*$
  • Theorem A.1: Test loss
  • Theorem A.1: OOD performance
  • ...and 36 more