Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalising E-prop to Deep Networks

Beren Millidge

TL;DR

The paper tackles credit assignment in deep recurrent networks and brain plausibility by extending E-prop from single-layer to arbitrarily deep architectures. It derives a depth-aware, online learning framework using two coupled traces across time and depth, and generalizes further to DAG computation graphs. The key contributions include the nested recursions for time and depth, yielding a linear-depth online update rule that preserves locality via eligibility traces. While theoretically appealing, the work remains preliminary with no empirical validation and discusses several practical caveats for scaling, such as multiple parameter groups and online versus end-of-episode updates. Overall, it advances brain-inspired online learning for deep recurrent networks and sets the stage for future experiments and refinements.

Abstract

Recurrent networks are typically trained with backpropagation through time (BPTT). However, BPTT requires storing the history of all states in the network and then replaying them sequentially backwards in time. This computation appears extremely implausible for the brain to implement. Real Time Recurrent Learning (RTRL) proposes an mathematically equivalent alternative where gradient information is propagated forwards in time locally alongside the regular forward pass, however it has significantly greater computational complexity than BPTT which renders it impractical for large networks. E-prop proposes an approximation of RTRL which reduces its complexity to the level of BPTT while maintaining a purely online forward update which can be implemented by an eligibility trace at each synapse. However, works on RTRL and E-prop ubiquitously investigate learning in a single layer with recurrent dynamics. However, learning in the brain spans multiple layers and consists of both hierarchal dynamics in depth as well as time. In this mathematical note, we extend the E-prop framework to handle arbitrarily deep networks, deriving a novel recursion relationship across depth which extends the eligibility traces of E-prop to deeper layers. Our results thus demonstrate an online learning algorithm can perform accurate credit assignment across both time and depth simultaneously, allowing the training of deep recurrent networks without backpropagation through time.

Generalising E-prop to Deep Networks

TL;DR

The paper tackles credit assignment in deep recurrent networks and brain plausibility by extending E-prop from single-layer to arbitrarily deep architectures. It derives a depth-aware, online learning framework using two coupled traces across time and depth, and generalizes further to DAG computation graphs. The key contributions include the nested recursions for time and depth, yielding a linear-depth online update rule that preserves locality via eligibility traces. While theoretically appealing, the work remains preliminary with no empirical validation and discusses several practical caveats for scaling, such as multiple parameter groups and online versus end-of-episode updates. Overall, it advances brain-inspired online learning for deep recurrent networks and sets the stage for future experiments and refinements.

Abstract

Recurrent networks are typically trained with backpropagation through time (BPTT). However, BPTT requires storing the history of all states in the network and then replaying them sequentially backwards in time. This computation appears extremely implausible for the brain to implement. Real Time Recurrent Learning (RTRL) proposes an mathematically equivalent alternative where gradient information is propagated forwards in time locally alongside the regular forward pass, however it has significantly greater computational complexity than BPTT which renders it impractical for large networks. E-prop proposes an approximation of RTRL which reduces its complexity to the level of BPTT while maintaining a purely online forward update which can be implemented by an eligibility trace at each synapse. However, works on RTRL and E-prop ubiquitously investigate learning in a single layer with recurrent dynamics. However, learning in the brain spans multiple layers and consists of both hierarchal dynamics in depth as well as time. In this mathematical note, we extend the E-prop framework to handle arbitrarily deep networks, deriving a novel recursion relationship across depth which extends the eligibility traces of E-prop to deeper layers. Our results thus demonstrate an online learning algorithm can perform accurate credit assignment across both time and depth simultaneously, allowing the training of deep recurrent networks without backpropagation through time.
Paper Structure (7 sections, 11 equations, 2 figures)

This paper contains 7 sections, 11 equations, 2 figures.

Table of Contents

  1. Preliminaries
  2. BPTT
  3. RTRL
  4. E-prop
  5. Extension to Deep Networks
  6. Extension to arbitrary DAGs
  7. Discussion

Figures (2)

  • Figure 1: Schematic computation graph of a single layer RNN model of the type generally used in the RTRL literature. The BPTT gradients are marked in red.
  • Figure 2: Schematic computation graph of our example two-layer 3-step RNN for which we derive the deep E-prop recursion.