AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
Rohan Mohapatra, Snehanshu Saha, Carlos A. Coello Coello, Anwesh Bhattacharya, Soma S. Dhavala, Sriparna Saha
TL;DR
AdaSwarm tackles neural network optimization by replacing exact gradients with EMPSO-based gradient approximations, enabling gradient-free training. The core idea is EMPSO, a PSO variant with exponentially weighted momentum, which yields gradient estimates that can substitute in backpropagation-like updates. The authors prove theoretical equivalences: GD corresponds to vanilla PSO gradients and SGD with momentum corresponds to EMPSO-derived gradients under suitable parameter mappings. Empirically, AdaSwarm achieves competitive or superior performance to Adam on diverse classification tasks and CNN benchmarks, often with faster convergence and support for non-differentiable losses such as MAE. The work suggests a practical path to broader loss-function choices and gradient-free optimization in deep learning.
Abstract
This paper introduces AdaSwarm, a novel gradient-free optimizer which has similar or even better performance than the Adam optimizer adopted in neural networks. In order to support our proposed AdaSwarm, a novel Exponentially weighted Momentum Particle Swarm Optimizer (EMPSO), is proposed. The ability of AdaSwarm to tackle optimization problems is attributed to its capability to perform good gradient approximations. We show that, the gradient of any function, differentiable or not, can be approximated by using the parameters of EMPSO. This is a novel technique to simulate GD which lies at the boundary between numerical methods and swarm intelligence. Mathematical proofs of the gradient approximation produced are also provided. AdaSwarm competes closely with several state-of-the-art (SOTA) optimizers. We also show that AdaSwarm is able to handle a variety of loss functions during backpropagation, including the maximum absolute error (MAE).