Gradient Estimation and Variance Reduction in Stochastic and Deterministic Models
Ronan Keane
TL;DR
This work develops and unifies gradient estimation for both stochastic and deterministic models via reverse differentiation, introducing a general RD framework that blends pathwise derivatives and score-function estimators. It analyzes convergence for stochastic gradient methods, demonstrates unbiased gradient estimation for diverse activation patterns and piecewise switching, and presents variance-reduction techniques (baselines, per-parameter baselines, and GAE-like schemes) to improve SGD performance. The framework is instantiated across stochastic differential equations, reinforcement learning, and traffic-flow applications, notably calibrating car-following models with an adjoint method and introducing a relaxation mechanism for lane-changing to reduce discontinuities and overreaction. The practical impact includes scalable gradient-based optimization for large, piecewise, and stochastic models, aided by differentiable traffic simulators (HAVSIM) and extensive empirical demonstrations of speed and accuracy gains over gradient-free methods. These results advance the ability to train complex models with stochastic components and discontinuities, enabling end-to-end calibration, control, and optimization in engineering and transportation systems, with broad implications for ML-assisted scientific discovery and simulation-based design.
Abstract
It seems that in the current age, computers, computation, and data have an increasingly important role to play in scientific research and discovery. This is reflected in part by the rise of machine learning and artificial intelligence, which have become great areas of interest not just for computer science but also for many other fields of study. More generally, there have been trends moving towards the use of bigger, more complex and higher capacity models. It also seems that stochastic models, and stochastic variants of existing deterministic models, have become important research directions in various fields. For all of these types of models, gradient-based optimization remains as the dominant paradigm for model fitting, control, and more. This dissertation considers unconstrained, nonlinear optimization problems, with a focus on the gradient itself, that key quantity which enables the solution of such problems. In chapter 1, we introduce the notion of reverse differentiation, a term which describes the body of techniques which enables the efficient computation of gradients. We cover relevant techniques both in the deterministic and stochastic cases. We present a new framework for calculating the gradient of problems which involve both deterministic and stochastic elements. In chapter 2, we analyze the properties of the gradient estimator, with a focus on those properties which are typically assumed in convergence proofs of optimization algorithms. Chapter 3 gives various examples of applying our new gradient estimator. We further explore the idea of working with piecewise continuous models, that is, models with distinct branches and if statements which define what specific branch to use.