Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory Networks: Towards Fully Biologically Plausible Learning

Jacobo Ruiz, Manas Gupta

TL;DR

This work tackles the challenge of designing biologically plausible, energy-efficient learning systems for vision tasks, addressing the gap where conventional neural networks rely on backpropagation and convolutions. The authors introduce the Memory Network, a single-pass, memory-based approach that updates a compact set of neuronal memories without backprop and without convolutions, enabling rapid learning and fast inference. Empirical results show strong performance on MNIST (up to 97% accuracy with tens of thousands of neurons) but only modest gains on CIFAR-10 (around 40% accuracy) due to dataset complexity, underscoring the need for further biologically plausible mechanisms and preprocessing strategies. Overall, the paper demonstrates a viable path toward more brain-like, energy-efficient AI and outlines concrete directions for improving representation learning under biologically constrained principles.

Abstract

The field of artificial intelligence faces significant challenges in achieving both biological plausibility and computational efficiency, particularly in visual learning tasks. Current artificial neural networks, such as convolutional neural networks, rely on techniques like backpropagation and weight sharing, which do not align with the brain's natural information processing methods. To address these issues, we propose the Memory Network, a model inspired by biological principles that avoids backpropagation and convolutions, and operates in a single pass. This approach enables rapid and efficient learning, mimicking the brain's ability to adapt quickly with minimal exposure to data. Our experiments demonstrate that the Memory Network achieves efficient and biologically plausible learning, showing strong performance on simpler datasets like MNIST. However, further refinement is needed for the model to handle more complex datasets such as CIFAR10, highlighting the need to develop new algorithms and techniques that closely align with biological processes while maintaining computational efficiency.

Memory Networks: Towards Fully Biologically Plausible Learning

TL;DR

This work tackles the challenge of designing biologically plausible, energy-efficient learning systems for vision tasks, addressing the gap where conventional neural networks rely on backpropagation and convolutions. The authors introduce the Memory Network, a single-pass, memory-based approach that updates a compact set of neuronal memories without backprop and without convolutions, enabling rapid learning and fast inference. Empirical results show strong performance on MNIST (up to 97% accuracy with tens of thousands of neurons) but only modest gains on CIFAR-10 (around 40% accuracy) due to dataset complexity, underscoring the need for further biologically plausible mechanisms and preprocessing strategies. Overall, the paper demonstrates a viable path toward more brain-like, energy-efficient AI and outlines concrete directions for improving representation learning under biologically constrained principles.

Abstract

The field of artificial intelligence faces significant challenges in achieving both biological plausibility and computational efficiency, particularly in visual learning tasks. Current artificial neural networks, such as convolutional neural networks, rely on techniques like backpropagation and weight sharing, which do not align with the brain's natural information processing methods. To address these issues, we propose the Memory Network, a model inspired by biological principles that avoids backpropagation and convolutions, and operates in a single pass. This approach enables rapid and efficient learning, mimicking the brain's ability to adapt quickly with minimal exposure to data. Our experiments demonstrate that the Memory Network achieves efficient and biologically plausible learning, showing strong performance on simpler datasets like MNIST. However, further refinement is needed for the model to handle more complex datasets such as CIFAR10, highlighting the need to develop new algorithms and techniques that closely align with biological processes while maintaining computational efficiency.
Paper Structure (26 sections, 4 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 26 sections, 4 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Criteria for Biological Plausibility
  4. Previous Work on Biologically Plausible Algorithms
  5. Similar algorithms to Memory Networks
  6. Methodology
  7. Biological Motivation and Design Choices
  8. Algorithm Overview
  9. Central Memory Structure: Neuron Tensor
  10. Processing of a training batch and neuron assignment
  11. Basic technique
  12. Technique using a Threshold
  13. Memory Update Step
  14. Distance Function Formulation
  15. Prediction Algorithm
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Update Mechanism with Full Memory: In this simplified example, the closest neuron to input1 is determined based on the computed distances (D1, D2, D3) to neurons with the same color label at memory state 0. The smallest distance, D2, indicates that input1 is closest to the neuron in row 6, leading to the latent representation r6 being updated with input1 and counter c6, as shown in memory state 1. This update process is conducted in parallel for the entire batch.
  • Figure 2: Accuracy vs. Number of Neurons for two models: one using a threshold and the other without. Fourteen different values for max_neurons were tested, with each configuration run five times. The error bars represent the standard deviation of accuracy. This figure corresponds to Experiments 7 and 8 from Table \ref{['tab:results_cifar10']}, illustrating that the accuracy plateaus as the number of neurons increases for both models.