Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dimension-free estimators of gradients of functions with(out) non-independent variables

Matieyendou Lamboni

TL;DR

This work addresses gradient computation when inputs are non-independent by introducing a unified stochastic framework that uses $\ell_p$-spherical distributions to form gradient surrogates. It defines dependent gradients via a tensor metric $G$ so that $grad(f)(\mathbf{x}) = G^{-1}(\mathbf{x}) \nabla f(\mathbf{x})$, and constructs $L$-point surrogates with estimators based on random vectors from $\mathcal{S}_{d,p}$. The authors prove dimension-free bias under appropriate smoothness and provide MSE rates that adapt to the regime of $p$ relative to $d$, achieving parametric or dimension-free rates with suitable choices of $p$ and bandwidth $h$. Numerical experiments on Rosenbrock and synthetic functions demonstrate competitive performance against finite-difference and Monte Carlo baselines, particularly when Gram-Schmidt decorrelation is employed. The results suggest scalable, dimension-aware gradient estimation for dependent inputs, with practical implications for zeroth-order optimization and derivative-free methods in high dimensions.

Abstract

This study proposes a unified stochastic framework for approximating and computing the gradient of every smooth function evaluated at non-independent variables, using $\ell_p$-spherical distributions on $\R^d$ with $d, p\geq 1$. The upper-bounds of the bias of the gradient surrogates do not suffer from the curse of dimensionality for any $p\geq 1$. Also, the mean squared errors (MSEs) of the gradient estimators are bounded by $K_0 N^{-1} d$ for any $p \in [1, 2]$, and by $K_1 N^{-1} d^{2/p}$ when $2 \leq p \ll d$ with $N$ the sample size and $K_0, K_1$ some constants. Taking $\max\left\{2, \log(d) \right\} < p \ll d$ allows for achieving dimension-free upper-bounds of MSEs. In the case where $d\ll p< +\infty$, the upper-bound $K_2 N^{-1} d^{2-2/p}/ (d+2)^2$ is reached with $K_2$ a constant. Such results lead to dimension-free MSEs of the proposed estimators, which boil down to estimators of the traditional gradient when the variables are independent. Numerical comparisons show the efficiency of the proposed approach.

Dimension-free estimators of gradients of functions with(out) non-independent variables

TL;DR

This work addresses gradient computation when inputs are non-independent by introducing a unified stochastic framework that uses -spherical distributions to form gradient surrogates. It defines dependent gradients via a tensor metric so that , and constructs -point surrogates with estimators based on random vectors from . The authors prove dimension-free bias under appropriate smoothness and provide MSE rates that adapt to the regime of relative to , achieving parametric or dimension-free rates with suitable choices of and bandwidth . Numerical experiments on Rosenbrock and synthetic functions demonstrate competitive performance against finite-difference and Monte Carlo baselines, particularly when Gram-Schmidt decorrelation is employed. The results suggest scalable, dimension-aware gradient estimation for dependent inputs, with practical implications for zeroth-order optimization and derivative-free methods in high dimensions.

Abstract

This study proposes a unified stochastic framework for approximating and computing the gradient of every smooth function evaluated at non-independent variables, using -spherical distributions on with . The upper-bounds of the bias of the gradient surrogates do not suffer from the curse of dimensionality for any . Also, the mean squared errors (MSEs) of the gradient estimators are bounded by for any , and by when with the sample size and some constants. Taking allows for achieving dimension-free upper-bounds of MSEs. In the case where , the upper-bound is reached with a constant. Such results lead to dimension-free MSEs of the proposed estimators, which boil down to estimators of the traditional gradient when the variables are independent. Numerical comparisons show the efficiency of the proposed approach.
Paper Structure (20 sections, 10 theorems, 57 equations, 4 figures, 7 tables)

This paper contains 20 sections, 10 theorems, 57 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Generalized surrogates of dependent gradients
  3. Notation
  4. Non-Euclidean or dependent gradients
  5. Stochastic surrogates of the dependent gradient and estimators
  6. Convergence analysis
  7. Dimension-free upper-bounds of the bias
  8. Mean squared errors of estimators of dependent gradients
  9. Computational issues
  10. Rosenbrock's function
  11. A synthetic function
  12. A synthetic function
  13. Conclusion
  14. Proof of Corollary \ref{['coro:parderord']}
  15. Proof of Lemma \ref{['lam:qthm']}
  16. ...and 5 more sections

Key Result

Corollary 1

Let $\mathbf{V}$ satisfying (eq:proned). Assume that $f \in \mathcal{H}_\alpha$ with $\alpha \in \{2, \ldots, 2L+1\}$ and $\beta_\ell$s are distinct. Then, there exists $\theta \in \{1, \ldots, L\}$ and reals coefficients $C_{1}, \ldots, C_{L}$ such that

Figures (4)

  • Figure 1: Average of error values against the sample sizes ($N$) for $p$-spherical distributions (i.e., $\mathbf{V}$) when $\mathbf{d=100}$. Panel (a) is obtained when $L=1$, while Panel (b) is associated with $L=2$.
  • Figure 2: Average of error values against the sample sizes ($N$) for $p$-balls (i.e., $\tilde{\mathbf{V}}$) and when $\mathbf{d=100}$. Panel (a) is obtained when $L=1$, while Panel (b) is associated with $L=2$.
  • Figure 3: Average of error values against the sample sizes ($N$) for $p$-spherical distributions (i.e., $\mathbf{V}$) when $\mathbf{d=1000}$. Panel (a) is obtained when $L=1$, while Panel (b) is associated with $L=2$.
  • Figure 4: Average of error values against the sample sizes ($N$) for $p$-balls (i.e., $\tilde{\mathbf{V}}$) and when $\mathbf{d=1000}$. Panel (a) is obtained when $L=1$, while Panel (b) is associated with $L=2$.

Theorems & Definitions (25)

  • Corollary 1
  • Remark 1
  • Corollary 2
  • proof
  • Remark 2
  • Lemma 1
  • proof
  • Corollary 3
  • proof
  • Remark 3
  • ...and 15 more