Learning in Log-Domain: Subthreshold Analog AI Accelerator Based on Stochastic Gradient Descent

TL;DR

The paper addresses energy-efficient on-device training for AI at the edge by proposing an analog accelerator for stochastic gradient descent with L2 regularization ($SGDr$) implemented in log-domain subthreshold CMOS with volatile memory. It develops a continuous-time formulation ($SGDr-CT$) and provides a circuit-level mapping of the learning rules to weight-learning nodes arranged in a crossbar, enabling in-memory training with minimal digital overhead. The architecture uses dual learning cells (positive and negative) and differential signaling, augmented by translinear log-domain multipliers and a geometric mean splitter to realize both forward and backward passes. Across AMS 0.35 µm simulations, the approach achieves a mean squared error below $0.87\%$ and effective 8-bit weight precision, indicating competitive accuracy with substantial potential energy and area benefits for edge training.

Abstract

The rapid proliferation of AI models, coupled with growing demand for edge deployment, necessitates the development of AI hardware that is both high-performance and energy-efficient. In this paper, we propose a novel analog accelerator architecture designed for AI/ML training workloads using stochastic gradient descent with L2 regularization (SGDr). The architecture leverages log-domain circuits in subthreshold MOS and incorporates volatile memory. We establish a mathematical framework for solving SGDr in the continuous time domain and detail the mapping of SGDr learning equations to log-domain circuits. By operating in the analog domain and utilizing weak inversion, the proposed design achieves significant reductions in transistor area and power consumption compared to digital implementations. Experimental results demonstrate that the architecture closely approximates ideal behavior, with a mean square error below 0.87% and precision as low as 8 bits. Furthermore, the architecture supports a wide range of hyperparameters. This work paves the way for energy-efficient analog AI hardware with on-chip training capabilities.

Figures (7)

• Figure 1: The basic concept of a memristive crossbar. The current accumulated at each column represents the output of a single artificial neuron before the activation function. Image: MovGP0 at the German-language Wikipedia, licensed under CC BY-SA 3.0, via Wikimedia Commons.
• Figure 2: Represents the discrete-time input feature $x[n]$ as a continuous-time step function $x(t)$.
• Figure 3: (a) Architecture of the proposed subthreshold CMOS analog accelerator, featuring a crossbar structure composed of weight-learning nodes. (b) Circuit-level realization of the weight-learning node, illustrating the integration of volatile memory and translinear MOS circuits for efficient weight updates.
• Figure 4: The estimated weights and loss curves of the ideal SGDr model and the circuit implementation (SGDr-CT) for five randomly generated datasets. The features were drawn from a uniform distribution (-1,1) with added Gaussian noise (scaled by 0.1 factor). The models were trained for 200 epochs with the following hyperparameters: $\lambda$ = 0.1, $\alpha$=$1e^{-3}$, $\Delta s$= 0.01 ms.
• Figure 5: Comparison of the accuracy of the ideal and circuit models for various hyperparameter values. The learning rate ($\alpha$) was set to $1 \times 10^{-2}$, $1 \times 10^{-3}$, and $1 \times 10^{-4}$, while the regularization coefficient ($\lambda$) was set to 0.2, 0.1, and 0.05.

Theorems & Definitions (2)

• proof
• proof