Convergence, design and training of continuous-time dropout as a random batch method
Antonio Álvarez-López, Martín Hernández
TL;DR
This paper develops a principled framework for continuous-time dropout using random batch methods (RBMs) in controlled differential equations arising from neural ODEs. By constructing an unbiased estimator that activates neuron batches over time with step $h$, the authors prove trajectory convergence at rate $O(h)$ and distribution stability in total variation at rate $O(\sqrt{h})$, along with adjoint-based bounds for training via Pontryagin’s Maximum Principle. The work also provides a design analysis linking batch schemes to convergence constants, showing that standard Bernoulli dropout is a special case and deriving a closed-form optimal $h$ through a cost–accuracy trade-off. Numerical experiments on a single-layer neural ODE validate the predicted rates, demonstrate regularization and performance gains, and reveal substantial memory and runtime savings with structured RBMs. Collectively, this framework offers rigorous, scalable guidance for deploying dropout-like regularization in continuous-time deep learning models and related measure-transport problems.
Abstract
We study dropout regularization in continuous-time models through the lens of random-batch methods -- a family of stochastic sampling schemes originally devised to reduce the computational cost of interacting particle systems. We construct an unbiased, well-posed estimator that mimics dropout by sampling neuron batches over time intervals of length $h$. Trajectory-wise convergence is established with linear rate in $h$ for the expected uniform error. At the distribution level, we establish stability for the associated continuity equation, with total-variation error of order $h^{1/2}$ under mild moment assumptions. During training with fixed batch sampling across epochs, a Pontryagin-based adjoint analysis bounds deviations in the optimal cost and control, as well as in gradient-descent iterates. On the design side, we compare convergence rates for canonical batch sampling schemes, recover standard Bernoulli dropout as a special case, and derive a cost--accuracy trade-off yielding a closed-form optimal $h$. We then specialize to a single-layer neural ODE and validate the theory on classification and flow matching, observing the predicted rates, regularization effects, and favorable runtime and memory profiles.