Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ADAM Optimization with Adaptive Batch Selection

Gyu Yeol Kim, Min-hwan Oh

TL;DR

AdamCB tackles inefficiencies in data sampling during Adam optimization by using combinatorial bandit sampling to form mini-batches. It provides a provable regret bound that improves over uniform-sampling Adam and corrected AdamBS, thanks to simultaneous multi-arm exploration-exploitation and unbiased gradient estimates. The approach leverages DepRound for efficient batch construction and an online-to-batch regret framework to guarantee convergence. Empirical results across MNIST, Fashion-MNIST, CIFAR-10, and larger models show consistent, faster convergence and superior performance compared to baseline Adam variants. Overall, AdamCB delivers both rigorous guarantees and practical gains for adaptive batch selection in Adam-based optimization.

Abstract

Adam is a widely used optimizer in neural network training due to its adaptive learning rate. However, because different data samples influence model updates to varying degrees, treating them equally can lead to inefficient convergence. To address this, a prior work proposed adapting the sampling distribution using a bandit framework to select samples adaptively. While promising, the bandit-based variant of Adam suffers from limited theoretical guarantees. In this paper, we introduce Adam with Combinatorial Bandit Sampling (AdamCB), which integrates combinatorial bandit techniques into Adam to resolve these issues. AdamCB is able to fully utilize feedback from multiple samples at once, enhancing both theoretical guarantees and practical performance. Our regret analysis shows that AdamCB achieves faster convergence than Adam-based methods including the previous bandit-based variant. Numerical experiments demonstrate that AdamCB consistently outperforms existing methods.

ADAM Optimization with Adaptive Batch Selection

TL;DR

AdamCB tackles inefficiencies in data sampling during Adam optimization by using combinatorial bandit sampling to form mini-batches. It provides a provable regret bound that improves over uniform-sampling Adam and corrected AdamBS, thanks to simultaneous multi-arm exploration-exploitation and unbiased gradient estimates. The approach leverages DepRound for efficient batch construction and an online-to-batch regret framework to guarantee convergence. Empirical results across MNIST, Fashion-MNIST, CIFAR-10, and larger models show consistent, faster convergence and superior performance compared to baseline Adam variants. Overall, AdamCB delivers both rigorous guarantees and practical gains for adaptive batch selection in Adam-based optimization.

Abstract

Adam is a widely used optimizer in neural network training due to its adaptive learning rate. However, because different data samples influence model updates to varying degrees, treating them equally can lead to inefficient convergence. To address this, a prior work proposed adapting the sampling distribution using a bandit framework to select samples adaptively. While promising, the bandit-based variant of Adam suffers from limited theoretical guarantees. In this paper, we introduce Adam with Combinatorial Bandit Sampling (AdamCB), which integrates combinatorial bandit techniques into Adam to resolve these issues. AdamCB is able to fully utilize feedback from multiple samples at once, enhancing both theoretical guarantees and practical performance. Our regret analysis shows that AdamCB achieves faster convergence than Adam-based methods including the previous bandit-based variant. Numerical experiments demonstrate that AdamCB consistently outperforms existing methods.

Paper Structure

This paper contains 57 sections, 19 theorems, 178 equations, 9 figures, 5 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations
  4. Expected Risk and Empirical Risk
  5. Expected Risk.
  6. Empirical Risk.
  7. Objective Function and Mini-Batches
  8. Objective Function.
  9. Mini-Batches.
  10. Regret Minimization
  11. Cumulative Regret.
  12. Online Regret.
  13. Related Work: Adam and Technical Issues in Convergence Guarantees
  14. Adam Optimizer
  15. Convergence of Adam
  16. ...and 42 more sections

Key Result

Theorem 1

Suppose Assumptions assum:bounded_gradient-assum:bounded_parameter_space hold, and we run $\texttt{AdamCB}$ for a total $T$ iterations with $\alpha_t=\frac{\alpha}{\sqrt{t}}$ and with $\beta_{1,t} := \beta_1 \lambda^{t-1}$, $\lambda \in (0,1)$. Then, the cumulative regret of $\texttt{AdamCB}$ (Algor

Figures (9)

  • Figure 1: Performances with MLP model on MNIST, Fashion MNIST, and CIFAR10
  • Figure 2: Performances with CNN model on MNIST, Fashion MNIST, and CIFAR10
  • Figure 3: Comparison of Adam-based optimizations on MLP model
  • Figure 4: Comparison of Adam-based optimizations on CNN model
  • Figure 5: Comparison of Adam-based optimizations on VGG model
  • ...and 4 more figures

Theorems & Definitions (20)

  • Theorem 1: Regret bound of $\texttt{AdamCB}$
  • Lemma 1
  • Lemma 2
  • Definition 1
  • Lemma 3
  • Lemma 4: Cauchy-Schwarz inequality
  • Lemma 5: Taylor series
  • Lemma 6: Upper bound for the harmonic series
  • Lemma 7
  • Lemma 8
  • ...and 10 more