Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning

Yupei Li, Manuel Milling, Björn W. Schuller

TL;DR

This paper surveys brain-inspired mechanisms—neurogenesis, neuroapoptosis, and neuroplasticity—and maps them to AI architectures, introducing dropin as a counterpart to dropout and arguing for lifelong learning in large models. It reviews architectural optimization approaches (e.g., lottery ticket hypothesis and student–teacher models) and discusses online, transfer, and reinforcement learning in dynamic environments. A taxonomy of artificial neurogenesis and neuroplasticity is presented, including dropin and numerous dropout and pruning variants, alongside a proposed neuroplasticity cycle for continual learning. The work highlights the mutual benefits and challenges of integrating neuroscience-inspired dynamics into AI, urging empirical validation and cross-disciplinary collaboration to advance adaptable, scalable models.

Abstract

Artificial Intelligence (AI) has achieved new levels of performance and spread in public usage with the rise of deep neural networks (DNNs). Initially inspired by human neurons and their connections, NNs have become the foundation of AI models for many advanced architectures. However, some of the most integral processes in the human brain, particularly neurogenesis and neuroplasticity in addition to the more spread neuroapoptosis have largely been ignored in DNN architecture design. Instead, contemporary AI development predominantly focuses on constructing advanced frameworks, such as large language models, which retain a static structure of neural connections during training and inference. In this light, we explore how neurogenesis, neuroapoptosis, and neuroplasticity can inspire future AI advances. Specifically, we examine analogous activities in artificial NNs, introducing the concepts of ``dropin'' for neurogenesis and revisiting ``dropout'' and structural pruning for neuroapoptosis. We additionally suggest neuroplasticity combining the two for future large NNs in ``life-long learning'' settings following the biological inspiration. We conclude by advocating for greater research efforts in this interdisciplinary domain and identifying promising directions for future exploration.

Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning

TL;DR

This paper surveys brain-inspired mechanisms—neurogenesis, neuroapoptosis, and neuroplasticity—and maps them to AI architectures, introducing dropin as a counterpart to dropout and arguing for lifelong learning in large models. It reviews architectural optimization approaches (e.g., lottery ticket hypothesis and student–teacher models) and discusses online, transfer, and reinforcement learning in dynamic environments. A taxonomy of artificial neurogenesis and neuroplasticity is presented, including dropin and numerous dropout and pruning variants, alongside a proposed neuroplasticity cycle for continual learning. The work highlights the mutual benefits and challenges of integrating neuroscience-inspired dynamics into AI, urging empirical validation and cross-disciplinary collaboration to advance adaptable, scalable models.

Abstract

Artificial Intelligence (AI) has achieved new levels of performance and spread in public usage with the rise of deep neural networks (DNNs). Initially inspired by human neurons and their connections, NNs have become the foundation of AI models for many advanced architectures. However, some of the most integral processes in the human brain, particularly neurogenesis and neuroplasticity in addition to the more spread neuroapoptosis have largely been ignored in DNN architecture design. Instead, contemporary AI development predominantly focuses on constructing advanced frameworks, such as large language models, which retain a static structure of neural connections during training and inference. In this light, we explore how neurogenesis, neuroapoptosis, and neuroplasticity can inspire future AI advances. Specifically, we examine analogous activities in artificial NNs, introducing the concepts of ``dropin'' for neurogenesis and revisiting ``dropout'' and structural pruning for neuroapoptosis. We additionally suggest neuroplasticity combining the two for future large NNs in ``life-long learning'' settings following the biological inspiration. We conclude by advocating for greater research efforts in this interdisciplinary domain and identifying promising directions for future exploration.

Paper Structure

This paper contains 30 sections, 1 equation, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Neuroplasticity in the Human Brain
  3. The Quest for Optimal NN configurations
  4. Duality of Small and Large Networks
  5. Lottery ticket hypothesis
  6. Student-teacher model
  7. Dynamic tasks solution
  8. Online Learning
  9. Transfer Learning
  10. Reinforcement Learning
  11. Neurogenesis, Neuroapoptosis, and Neuroplasticity in AI
  12. Neurogenesis: proliferation and dropin
  13. Dropin
  14. Model Expansion
  15. Adaptive Learning
  16. ...and 15 more sections

Figures (1)

  • Figure 1: Neuroplasticity, neurogenesis, and neuroapoptosis relationship. Neuroplasticity is the comprehensive mechanism encompassing both neurogenesis and neuroapoptosis.