Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization-Memorization Machines

Zhen Wang, Yuan-Hai Shao

TL;DR

The paper introduces a generalization-memorization mechanism to enable memorization of training data without sacrificing generalization. By defining a memory cost function and a memory influence function, it augments standard learning decisions to form hard and soft generalization-memorization machines (HGMM/SGMM), which are solvable as quadratic programs. The framework unifies and extends SVM-based memorization, showing that SVM^m is a special case of SGMM, with clear geometric and optimization interpretations. Empirical results demonstrate that HGMM can achieve zero empirical risk with strong generalization on many datasets, while SGMM offers robustness to label noise and competitive performance, suggesting practical value for small-data or noisy settings.

Abstract

Classifying the training data correctly without over-fitting is one of the goals in machine learning. In this paper, we propose a generalization-memorization mechanism, including a generalization-memorization decision and a memory modeling principle. Under this mechanism, error-based learning machines improve their memorization abilities of training data without over-fitting. Specifically, the generalization-memorization machines (GMM) are proposed by applying this mechanism. The optimization problems in GMM are quadratic programming problems and could be solved efficiently. It should be noted that the recently proposed generalization-memorization kernel and the corresponding support vector machines are the special cases of our GMM. Experimental results show the effectiveness of the proposed GMM both on memorization and generalization.

Generalization-Memorization Machines

TL;DR

The paper introduces a generalization-memorization mechanism to enable memorization of training data without sacrificing generalization. By defining a memory cost function and a memory influence function, it augments standard learning decisions to form hard and soft generalization-memorization machines (HGMM/SGMM), which are solvable as quadratic programs. The framework unifies and extends SVM-based memorization, showing that SVM^m is a special case of SGMM, with clear geometric and optimization interpretations. Empirical results demonstrate that HGMM can achieve zero empirical risk with strong generalization on many datasets, while SGMM offers robustness to label noise and competitive performance, suggesting practical value for small-data or noisy settings.

Abstract

Classifying the training data correctly without over-fitting is one of the goals in machine learning. In this paper, we propose a generalization-memorization mechanism, including a generalization-memorization decision and a memory modeling principle. Under this mechanism, error-based learning machines improve their memorization abilities of training data without over-fitting. Specifically, the generalization-memorization machines (GMM) are proposed by applying this mechanism. The optimization problems in GMM are quadratic programming problems and could be solved efficiently. It should be noted that the recently proposed generalization-memorization kernel and the corresponding support vector machines are the special cases of our GMM. Experimental results show the effectiveness of the proposed GMM both on memorization and generalization.
Paper Structure (12 sections, 4 theorems, 25 equations, 1 figure, 5 tables)

This paper contains 12 sections, 4 theorems, 25 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Introduction
  3. Review of SVM and SVM$^m$
  4. Generalization-memorization mechanism
  5. Memory cost function
  6. Memory influence function
  7. Memory model principle
  8. Generalization-Memorization Machines
  9. Hard and soft generalization-memorization machines
  10. Discussion
  11. Experiments
  12. Conclusions

Key Result

Theorem 5.1

The empirical risk of HGMM is zero if and only if there is at least a feasible solution to problem Model2mat.

Figures (1)

  • Figure 1: A toy example to show the memorization ability of HGMM and SGMM, where memory influence function \ref{['Influencefunction1']} and $K({\bf x}_i,{\bf x}_j)={\bf x}_i^\top{\bf x}_j$ are hired in problems \ref{['Model1dual']} and \ref{['Modelweakdual']}.

Theorems & Definitions (4)

  • Theorem 5.1
  • Theorem 5.2
  • Theorem 5.3
  • Theorem 5.4