Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multiplicative Learning

Han Kim, Hyungjoon Soh, Vipul Periwal, Junghyo Jo

TL;DR

The paper addresses the challenge of efficiently training deep neural networks by introducing Expectation Reflection (ER), a multiplicative, ratio-based weight-update rule that eliminates the need for loss functions and learning-rate hyperparameters. By connecting ER to gradient descent and target propagation, the authors present a multilayer ER algorithm that updates weights in a single iteration and interpret ER as a form of TP with an inverse-like backward step. Empirical results on MNIST and CIFAR-10 demonstrate that multilayer ER can achieve competitive performance with very few weight updates, notably reaching near-peak performance after a single update on MNIST without tuning hyperparameters. The work suggests ER as a scalable, biologically plausible alternative to backpropagation, with potential implications for fast, autonomous training in neural networks.

Abstract

Efficient training of artificial neural networks remains a key challenge in deep learning. Backpropagation (BP), the standard learning algorithm, relies on gradient descent and typically requires numerous iterations for convergence. In this study, we introduce Expectation Reflection (ER), a novel learning approach that updates weights multiplicatively based on the ratio of observed to predicted outputs. Unlike traditional methods, ER maintains consistency without requiring ad hoc loss functions or learning rate hyperparameters. We extend ER to multilayer networks and demonstrate its effectiveness in performing image classification tasks. Notably, ER achieves optimal weight updates in a single iteration. Additionally, we reinterpret ER as a modified form of gradient descent incorporating the inverse mapping of target propagation. These findings suggest that ER provides an efficient and scalable alternative for training neural networks.

Multiplicative Learning

TL;DR

The paper addresses the challenge of efficiently training deep neural networks by introducing Expectation Reflection (ER), a multiplicative, ratio-based weight-update rule that eliminates the need for loss functions and learning-rate hyperparameters. By connecting ER to gradient descent and target propagation, the authors present a multilayer ER algorithm that updates weights in a single iteration and interpret ER as a form of TP with an inverse-like backward step. Empirical results on MNIST and CIFAR-10 demonstrate that multilayer ER can achieve competitive performance with very few weight updates, notably reaching near-peak performance after a single update on MNIST without tuning hyperparameters. The work suggests ER as a scalable, biologically plausible alternative to backpropagation, with potential implications for fast, autonomous training in neural networks.

Abstract

Efficient training of artificial neural networks remains a key challenge in deep learning. Backpropagation (BP), the standard learning algorithm, relies on gradient descent and typically requires numerous iterations for convergence. In this study, we introduce Expectation Reflection (ER), a novel learning approach that updates weights multiplicatively based on the ratio of observed to predicted outputs. Unlike traditional methods, ER maintains consistency without requiring ad hoc loss functions or learning rate hyperparameters. We extend ER to multilayer networks and demonstrate its effectiveness in performing image classification tasks. Notably, ER achieves optimal weight updates in a single iteration. Additionally, we reinterpret ER as a modified form of gradient descent incorporating the inverse mapping of target propagation. These findings suggest that ER provides an efficient and scalable alternative for training neural networks.

Paper Structure

This paper contains 14 sections, 38 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Backpropagation
  4. Target Propagation
  5. Expectation Reflection
  6. Single-Layer Expectation Reflection
  7. Expectation Reflection as Gradient Descent
  8. Target Propagation in Expectation Reflection
  9. Multilayer Expectation Reflection
  10. Formulation
  11. Algorithm
  12. Results
  13. Conclusion
  14. Appendix: Mini-batch training

Figures (1)

  • Figure 1: Learning trajectories of the multilayer Expectation Reflection method. Training errors (blue circles) and test errors (orange squares) for the MNIST dataset (left) and CIFAR-10 dataset (right) are plotted against learning epochs.