Dimensional Peeking for Low-Variance Gradients in Zeroth-Order Discrete Optimization via Simulation

TL;DR

Dimensional peeking introduces a variance-reduction technique for zeroth-order optimization over discrete simulations by lifting perturbations to control-flow-equivalence classes, yielding unbiased gradient estimates relative to a smoothed objective. The method, implemented via a perturbed arithmetic type in C++, groups perturbations that traverse the same control-flow path and weights them by their probabilities, boosting information per evaluation. Empirical results across CityFlow, Hotel, and DynamNews show variance reductions up to 7.9x with acceptable overhead, translating into faster and more reliable convergence than several meta-heuristics, particularly for non-convex discrete problems. The approach broadens the practicality of gradient-based SBO and suggests potential extensions to trust-region methods and reinforcement learning with discrete actions.

Abstract

Gradient-based optimization methods are commonly used to identify local optima in high-dimensional spaces. When derivatives cannot be evaluated directly, stochastic estimators can provide approximate gradients. However, these estimators' perturbation-based sampling of the objective function introduces variance that can lead to slow convergence. In this paper, we present dimensional peeking, a variance reduction method for gradient estimation in discrete optimization via simulation. By lifting the sampling granularity from scalar values to classes of values that follow the same control flow path, we increase the information gathered per simulation evaluation. Our derivation from an established smoothed gradient estimator shows that the method does not introduce any bias. We present an implementation via a custom numerical data type to transparently carry out dimensional peeking over C++ programs. Variance reductions by factors of up to 7.9 are observed for three simulation-based optimization problems with high-dimensional input. The optimization progress compared to three meta-heuristics shows that dimensional peeking increases the competitiveness of zeroth-order optimization for discrete and non-convex simulations.

Figures (4)

• Figure 1: Dimensional peeking in a two-dimensional space. A primal perturbed input (black circle) is augmented by all alternative values of non-negligible probability per dimension. By identifying and grouping input values that follow the same path at branches (triangles), the sampling granularity is lifted from scalars to control flow-equivalent classes.
• Figure 2: Dimensional peeking for a program comprised of arithmetic and a conditional branch, evaluated at the primal perturbed input $x = [2\ 1\ 7]$. Numbers in brackets represent the variable values under different perturbations, with the primal value shown in bold. Subscripts indicate the perturbed decision variables the value depends on. Evaluating the conditional statement if y < 20 rules out values on two of the three dimensions (crossed-out numbers).
• Figure 3: Comparison of the optimization progress over time between PGO and PGO-DP with $\sigma = 1$ and learning rates of $0.01$ and $0.1$ for SGD and Adam. For Hotel and DynamNews, PGO-DP significantly outperforms PGO.
• Figure 4: Optimization progress over time with the best hyperparametrization for each method. Gradient descent via PGO and PGO-DP excels at Hotel and is competitive at DynamNews. PGO-DP consistently outperforms PGO.