Mini-batch descent in semiflows

TL;DR

The paper develops a continuous mini-batch descent framework for gradient flows of convex lower semicontinuous functionals on Hilbert spaces, introducing a time-discretized, randomized batching mechanism with waiting time $\varepsilon$. It proves that the mini-batch trajectory $v_{\varepsilon}$ converges in expectation to the full gradient flow $u$ under a non-biased subgradient decomposition and a variance measure $\Lambda$, with explicit bounds and rates that improve under Lipschitz/Hölder assumptions. A randomized minimizing movement scheme is also analyzed, yielding convergence to the gradient flow and, under inf-compactness, convergence to minimizers of the full loss; the framework is extended to a variety of problems, including constrained optimization, sparse inversion, and random domain decomposition for parabolic obstacle problems, with several illustrative numerical examples. The results provide a versatile, variance-aware approach to trajectory-level optimization in infinite-dimensional settings, enabling efficient approximations of complex evolution equations and PDEs with practical gains in computation and robustness to stochastic subproblem selection.

Abstract

This paper investigates the application of mini-batch gradient descent to semiflows (gradient flows). Given a loss function (potential), we introduce a continuous version of mini-batch gradient descent by randomly selecting sub-loss functions over time, defining a piecewise flow. We prove that, under suitable assumptions on the potential generating the semiflow, the \textit{mini-batch descent flow} trajectory closely approximates the original semiflow trajectory on average. In addition, we study a randomized minimizing movement scheme that also approximates the semiflow of the full loss function. We illustrate the versatility of this approach across various problems, including constrained optimization, sparse inversion, and domain decomposition. Finally, we validate our results with several numerical examples.

Figures (9)

• Figure 1: (A) Illustration of the cost function for $(u_1, u_2) \in [6, 10] \times [4, 18]$. We can also observe the feasible set (and its projection onto $\mathbb{R}^2$), illustrated with a darker color. The star denotes the minimum of the functional in the feasible set. (B) Trajectories of the gradient flow for different initial points. (C) Projection of the feasible set onto the plane, along with the trajectories of the gradient flow. The contour lines of the functional are also illustrated.
• Figure 2: (A) Thin lines show different realizations of the mini-batch descent flow. The dashed line illustrates the average of the different realizations to approximate the average mini-batch descent flow. The solid line shows the previously calculated gradient flow. (B) We illustrate the projection onto the $\mathbb R^2$ plane of the trajectories mentioned in (A).
• Figure 3: We illustrate the convergence as $K\rightarrow \infty$ (equivalent to $\varepsilon\rightarrow0$) of the mini-batch descent flow.
• Figure 4: (A) Illustration of the gradient descent trajectories defined by system \ref{['eq:gf_sparse']}. The black star denotes the minimum of $\Phi$. (B) Each curve corresponds to a different coordinate of the trajectory. Horizontal lines mark the optimal of $\Phi$.).
• Figure 5: (A) Illustration of the expected value of the mini-batch descent flow trajectory. This trajectory star is from the same value of the gradient flow. (B) Illustration of the mini-batch descent flow. Thin curves represent different realizations for each coordinate, while thick lines indicate the average outcomes of the mini-batch descent flow.

Theorems & Definitions (49)

• Theorem 2.1
• proof
• Proposition 2.2
• proof
• Theorem 2.3
• proof
• Theorem 2.4
• proof
• Theorem 2.5
• proof