Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A universal approximation theorem for nonlinear resistive networks

Benjamin Scellier, Siddhartha Mishra

TL;DR

This work establishes a universal function approximation theorem for nonlinear resistive networks built from ideal voltage sources, linear resistors, diodes, and voltage-controlled voltage sources. The authors prove that deep resistive networks (DRNs) can approximate ReLU neural networks to arbitrary accuracy, and since ReLU nets are universal on compact sets, DRNs are universal approximators as well. The result provides a constructive pathway to translate a ReLU NN into an approximately equivalent DRN, bridging analog resistor Network computation with established neural network universality. Simulations on standard datasets corroborate the theoretical claims and illuminate practical considerations, such as the need for large initial gains and the potential of bidirectional amplifiers, while acknowledging idealization caveats.

Abstract

Resistor networks have recently been studied as analog computing platforms for machine learning, particularly due to their compatibility with the Equilibrium Propagation training framework. In this work, we explore the computational capabilities of these networks. We prove that electrical networks consisting of voltage sources, linear resistors, diodes, and voltage-controlled voltage sources (VCVSs) can approximate any continuous function to arbitrary precision. Central to our proof is a method for translating a neural network with rectified linear units into an approximately equivalent electrical network comprising these four elements. Our proof relies on two assumptions: (a) that circuit elements are ideal, and (b) that variable resistor conductances and VCVS amplification factors can take any value (arbitrarily small or large). Our findings provide insights that could guide the development of universal self-learning electrical networks.

A universal approximation theorem for nonlinear resistive networks

TL;DR

This work establishes a universal function approximation theorem for nonlinear resistive networks built from ideal voltage sources, linear resistors, diodes, and voltage-controlled voltage sources. The authors prove that deep resistive networks (DRNs) can approximate ReLU neural networks to arbitrary accuracy, and since ReLU nets are universal on compact sets, DRNs are universal approximators as well. The result provides a constructive pathway to translate a ReLU NN into an approximately equivalent DRN, bridging analog resistor Network computation with established neural network universality. Simulations on standard datasets corroborate the theoretical claims and illuminate practical considerations, such as the need for large initial gains and the potential of bidirectional amplifiers, while acknowledging idealization caveats.

Abstract

Resistor networks have recently been studied as analog computing platforms for machine learning, particularly due to their compatibility with the Equilibrium Propagation training framework. In this work, we explore the computational capabilities of these networks. We prove that electrical networks consisting of voltage sources, linear resistors, diodes, and voltage-controlled voltage sources (VCVSs) can approximate any continuous function to arbitrary precision. Central to our proof is a method for translating a neural network with rectified linear units into an approximately equivalent electrical network comprising these four elements. Our proof relies on two assumptions: (a) that circuit elements are ideal, and (b) that variable resistor conductances and VCVS amplification factors can take any value (arbitrarily small or large). Our findings provide insights that could guide the development of universal self-learning electrical networks.
Paper Structure (32 sections, 9 theorems, 76 equations, 4 figures, 4 tables)

This paper contains 32 sections, 9 theorems, 76 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Nonlinear Resistive Networks
  3. Universal Approximation Through Neural Network Approximation
  4. ReLU Neural Network
  5. Deep Resistive Network
  6. Approximating a ReLU Neural Network With a DRN
  7. Simulations
  8. Discussion
  9. Proofs
  10. Proof of Lemma \ref{['lma:deep-resistive-network']}
  11. Proof of Lemma \ref{['lma:maximum-principle']}
  12. Proof of Theorem \ref{['thm:drn-approximate-relu-nn']}
  13. Proof of Theorem \ref{['thm:nrn-universal']}
  14. Bidirectional Amplifiers
  15. Simulation Details
  16. ...and 17 more sections

Key Result

Theorem 1

Given any continuous function $f : \mathbb{R}^q \to \mathbb{R}^r$, and given any compact subset $C \subset \mathbb{R}^q$ and any $\epsilon>0$, there exists a nonlinear resistive network with $q$ input voltage sources and $r$ output branches such that, under the above assumptions of ideality, the fun

Figures (4)

  • Figure 1: A nonlinear resistive network composed of variable resistors, voltage sources, diodes and voltage-controlled voltage sources (VCVSs). Voltage sources are used as inputs ($x$) and voltages across pairs of 'output nodes' are used as outputs ($y$). Variable resistors represent the trainable weights, diodes introduce nonlinearities, and the VCVSs are used for amplification. Such electrical networks are universal function approximators (Theorem \ref{['thm:nrn-universal']}). Note: for simplicity, the input terminals of the VCVS are not represented on the figure ; only its output terminals are displayed.
  • Figure 2: Current-voltage (i-v) characteristics of ideal elements. A linear resistor is characterized by Ohm's law: $i = g v$, where $g$ is the conductance of the resistor ($g=1/r$ where $r$ is the resistance). An ideal diode is characterized by $i=0$ for $v \leq 0$ ("off state", behaving like an open switch), and $v=0$ for $i>0$ ("on state", behaving like a closed switch). An ideal voltage source is characterized by $v=v_0$ for some constant voltage $v_0$. Finally, a voltage-controlled voltage source (VCVS) has four terminals -- two terminals that form an input voltage $v_{\rm in}$ and two terminals that form an output voltage $v_{\rm out}$ -- and is characterized by the relationship $v_{\rm out} = A \; v_{\rm in}$, where $A>0$ is the amplification factor, or gain. Note: for simplicity, the input terminals of the VCVS are not represented on the figure ; only its output terminals are represented.
  • Figure 3: A deep resistive network (DRN).(Top) A DRN consists of 'units' (excitatory, inhibitory or linear, shown in blue) interconnected by variable resistors (in red). Input voltages ($x_1$ and $x_2$) are applied to the network via voltage sources (not shown in the figure for readability) and are amplified using VCVSs to generate the electrical potentials $+A^{(0)}x_1$, $-A^{(0)}x_1$, $+A^{(0)}x_2$, and $-A^{(0)}x_2$ at designated 'input nodes' (in purple). The model's prediction is read from a set of output nodes ($v_1^{(3)}$, $v_2^{(3)}$, $v_3^{(3)}$, and $v_4^{(3)}$). Additionally, each layer includes two voltage sources serving as 'biases' (shown in black). (Bottom) Each unit is connected to ground through a resistor. Nonlinear units are constructed by placing a diode between the unit's node and ground. Depending on the orientation of the diode, nonlinear units can be either 'excitatory' or 'inhibitory'. Units without a diode are called 'linear' units.
  • Figure 4: Top. Bidirectional amplifier with gain $a$. Middle. A unit is composed of a bidirectional amplifier (with amplification factor $a$), followed by a resistor and a diode between the unit's node and ground. Depending on the orientation of the diode, units come in two flavors: excitatory units and inhibitory units. Bottom. A DRN with bidirectional amplifiers.

Theorems & Definitions (13)

  • Theorem 1: A universal approximation theorem for nonlinear resistive networks
  • Theorem 2: ReLU neural networks are universal function approximators
  • Lemma 2: Equations of a DRN
  • Lemma 2: A maximum principle for DRNs
  • Theorem 3: Approximation of a ReLU neural network with a DRN
  • Lemma 3: Equations of a DRN
  • proof : Proof of Lemma \ref{['lma:deep-resistive-network']}
  • Lemma 3: A maximum principle for DRNs
  • proof : Proof of Lemma \ref{['lma:maximum-principle']}
  • Theorem 3: Approximation of a ReLU neural network with a DRN
  • ...and 3 more