Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization in Neural Networks: A Broad Survey

Chris Rohlfs

TL;DR

The paper proposes a taxonomy of generalization for neural networks across six abstraction axes: Samples, Distributions, Domains, Tasks, Modalities, and Scopes. It surveys theoretical and empirical results on overfitting, cross-distribution generalization, and domain transfer, highlighting regularization, data augmentation, and transfer learning as core strategies, and discusses causal and counterfactual perspectives. It also reviews task-generalization advances (few-shot learning, transformers), cross-modality work (vision-language, multimodal models), and scope generalization through knowledge graphs and explainability tools, drawing insights from neuroscience to motivate modular, hierarchical designs. The work emphasizes foundation models and neurosymbolic approaches as promising paths toward stable, abstract reasoning in AI with broad practical impact across vision, language, and multimodal tasks.

Abstract

This paper reviews concepts, modeling approaches, and recent findings along a spectrum of different levels of abstraction of neural network models including generalization across (1) Samples, (2) Distributions, (3) Domains, (4) Tasks, (5) Modalities, and (6) Scopes. Strategies for (1) sample generalization from training to test data are discussed, with suggestive evidence presented that, at least for the ImageNet dataset, popular classification models show substantial overfitting. An empirical example and perspectives from statistics highlight how models' (2) distribution generalization can benefit from consideration of causal relationships and counterfactual scenarios. Transfer learning approaches and results for (3) domain generalization are summarized, as is the wealth of domain generalization benchmark datasets available. Recent breakthroughs surveyed in (4) task generalization include few-shot meta-learning approaches and the emergence of transformer-based foundation models such as those used for language processing. Studies performing (5) modality generalization are reviewed, including those that integrate image and text data and that apply a biologically-inspired network across olfactory, visual, and auditory modalities. Higher-level (6) scope generalization results are surveyed, including graph-based approaches to represent symbolic knowledge in networks and attribution strategies for improving networks' explainability. Additionally, concepts from neuroscience are discussed on the modular architecture of brains and the steps by which dopamine-driven conditioning leads to abstract thinking.

Generalization in Neural Networks: A Broad Survey

TL;DR

The paper proposes a taxonomy of generalization for neural networks across six abstraction axes: Samples, Distributions, Domains, Tasks, Modalities, and Scopes. It surveys theoretical and empirical results on overfitting, cross-distribution generalization, and domain transfer, highlighting regularization, data augmentation, and transfer learning as core strategies, and discusses causal and counterfactual perspectives. It also reviews task-generalization advances (few-shot learning, transformers), cross-modality work (vision-language, multimodal models), and scope generalization through knowledge graphs and explainability tools, drawing insights from neuroscience to motivate modular, hierarchical designs. The work emphasizes foundation models and neurosymbolic approaches as promising paths toward stable, abstract reasoning in AI with broad practical impact across vision, language, and multimodal tasks.

Abstract

This paper reviews concepts, modeling approaches, and recent findings along a spectrum of different levels of abstraction of neural network models including generalization across (1) Samples, (2) Distributions, (3) Domains, (4) Tasks, (5) Modalities, and (6) Scopes. Strategies for (1) sample generalization from training to test data are discussed, with suggestive evidence presented that, at least for the ImageNet dataset, popular classification models show substantial overfitting. An empirical example and perspectives from statistics highlight how models' (2) distribution generalization can benefit from consideration of causal relationships and counterfactual scenarios. Transfer learning approaches and results for (3) domain generalization are summarized, as is the wealth of domain generalization benchmark datasets available. Recent breakthroughs surveyed in (4) task generalization include few-shot meta-learning approaches and the emergence of transformer-based foundation models such as those used for language processing. Studies performing (5) modality generalization are reviewed, including those that integrate image and text data and that apply a biologically-inspired network across olfactory, visual, and auditory modalities. Higher-level (6) scope generalization results are surveyed, including graph-based approaches to represent symbolic knowledge in networks and attribution strategies for improving networks' explainability. Additionally, concepts from neuroscience are discussed on the modular architecture of brains and the steps by which dopamine-driven conditioning leads to abstract thinking.
Paper Structure (28 sections, 18 figures, 12 tables)

This paper contains 28 sections, 18 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Sample Generalization
  3. Bias and Variance
  4. Conceptual Framework
  5. Empirical Strategies
  6. Application to ImageNet
  7. Distribution Generalization
  8. Broadening Out-of-Sample Performance
  9. Counterfactuals and Causality
  10. Interaction Effects
  11. Domain Generalization
  12. Concept Drift
  13. Observable Shift
  14. Transfer Learning
  15. Empirical Results
  16. ...and 13 more sections

Figures (18)

  • Figure 1: Types of Generalization
  • Figure 2: Bias-Variance Tradeoff in Simulated Data
  • Figure 3: Error Rates versus Complexity on the ImageNet Data
  • Figure 4: Example In- and Out-of-Distribution Test Cases
  • Figure 5: Concept Drift
  • ...and 13 more figures