A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation

TL;DR

This paper formalizes a unified view of optimization in machine learning and neuroscience by organizing methods across derivative orders, from first-order gradient descent to zeroth-order probing. It surveys gradient-based techniques (including Newton, natural gradient, and modern curvature approximations), derivative-free strategies (direct search, stochastic metaheuristics, model-based surrogates, and population methods), and their adaptations to neural networks, highlighting scalability, sample efficiency, and the role of curvature. It then argues that biological learning can be understood through a zeroth-order lens, where intrinsic neural noise acts as probing and global modulators provide reinforcement, connecting learning in the brain to optimization principles and offering guidance for neuromorphic hardware. The synthesis elucidates how stochasticity and local learning with global feedback can yield robust, scalable learning in both artificial and natural systems, suggesting practical pathways for energy-efficient AI and deeper insights into brain function.

Abstract

Iterative optimization is central to modern artificial intelligence (AI) and provides a crucial framework for understanding adaptive systems. This review provides a unified perspective on this subject, bridging classic theory with neural network training and biological learning. Although gradient-based methods, powered by the efficient but biologically implausible backpropagation (BP), dominate machine learning, their computational demands can hinder scalability in high-dimensional settings. In contrast, derivative-free or zeroth-order (ZO) optimization feature computationally lighter approaches that rely only on function evaluations and randomness. While generally less sample efficient, recent breakthroughs demonstrate that modern ZO methods can effectively approximate gradients and achieve performance competitive with BP in neural network models. This ZO paradigm is also particularly relevant for biology. Its core principles of random exploration (probing) and feedback-guided adaptation (reinforcing) parallel key mechanisms of biological learning, offering a mathematically principled perspective on how the brain learns. In this review, we begin by categorizing optimization approaches based on the order of derivative information they utilize, ranging from first-, second-, and higher-order gradient-based to ZO methods. We then explore how these methods are adapted to the unique challenges of neural network training and the resulting learning dynamics. Finally, we build upon these insights to view biological learning through an optimization lens, arguing that a ZO paradigm leverages the brain's intrinsic noise as a computational resource. This framework not only illuminates our understanding of natural intelligence but also holds vast implications for neuromorphic hardware, helping us design fast and energy-efficient AI systems that exploit intrinsic hardware noise.

Figures (3)

• Figure 1: A visual guide to optimization methods.(a) Objective function landscape depicting example trajectories for different categories of methods. The higher the order, the more information it considers, and the fewer updates it needs to find the minimum. (b) Taxonomy tree categorizing the broad groups of optimization methods discussed in the review.
• Figure 2: Learning dynamics in neural networks.(a) Overview of the computational graph and operations of backpropagation in an $L$-layer neural network. $E$ represents the output error between the network output $\hat{y}$ and a given target, $\delta_l^{(i)}$ is the node-specific error of layer $l$ and node $i$, $\theta_{l}^{(i)}$ represents a single parameter in the network in layer $l$ connecting to node $i$, $\varphi$ is an activation function, $h_i$ represent the nodes outputs and $a_i$ represent the pre-activations. Only some of the values of $\theta$ and $\delta$ are shown for simplification. (b) Loss landscapes and how different minima affect generalization. (c) Double descent and grokking phenomena.
• Figure 3: Optimization and learning in biology.(a) Overview of the core high-level elements involved in biological learning. Cortical neurons interact via synapses $w_{ij}$, whose efficacy is instantaneously perturbed by synaptic noise $\xi_{ij}$ --this noise can also be conceived as spontaneous neural firing. A global feedback signal $\delta$ is then delivered by a modulatory neuron (e.g., dopaminergic), which signals changes in performance or expectations (e.g., reward prediction error). Synaptic changes are mediated by eligibility traces $e_{ij}$, a local memory of past activity correlations. (b) Computational mapping to zeroth-order optimization principles. The first formula illustrates how the probing mechanism is integrated into the processing pathways. The second formula shows how each synapse is tagged with an eligibility trace, capturing local sensitivity and activity correlations. The final formula provides the reinforcing mechanism, estimating a descent direction based on changes in performance or expectation.