Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

It's all the (Exponential) Family: An Equivalence between Maximum Likelihood Estimation and Control Variates for Sketching Algorithms

Keegan Kang, Kerong Wang, Ding Zhang, Rameshwar Pratap, Bhisham Dev Verma, Benedict H. W. Wong

TL;DR

This work addresses variance reduction and reproducibility in sketching-based estimators by establishing an equivalence between maximum likelihood estimation (MLE) and control variate estimators (CVE) within exponential families. It proves that, under suitable conditions, the asymptotic variance reductions of MLE and CVE are identical, which yields an EM-style CVE algorithm (CV-EM) that converges to the MLE and preserves the theoretical variance. Empirically, CV-EM demonstrates faster convergence and greater numerical stability than standard root-finding methods for the bivariate Normal distribution, and the approach extends to Hutchinson-type trace estimators by enabling MLE-like updates through CV weights. The results offer a unifying, reproducible framework for evaluating and computing estimators that blend MLE and CVE in sketching contexts, with practical heuristics for when CV weights are known.

Abstract

Maximum likelihood estimators (MLE) and control variate estimators (CVE) have been used in conjunction with known information across sketching algorithms and applications in machine learning. We prove that under certain conditions in an exponential family, an optimal CVE will achieve the same asymptotic variance as the MLE, giving an Expectation-Maximization (EM) algorithm for the MLE. Experiments show the EM algorithm is faster and numerically stable compared to other root finding algorithms for the MLE for the bivariate Normal distribution, and we expect this to hold across distributions satisfying these conditions. We show how the EM algorithm leads to reproducibility for algorithms using MLE / CVE, and demonstrate how the EM algorithm leads to finding the MLE when the CV weights are known.

It's all the (Exponential) Family: An Equivalence between Maximum Likelihood Estimation and Control Variates for Sketching Algorithms

TL;DR

This work addresses variance reduction and reproducibility in sketching-based estimators by establishing an equivalence between maximum likelihood estimation (MLE) and control variate estimators (CVE) within exponential families. It proves that, under suitable conditions, the asymptotic variance reductions of MLE and CVE are identical, which yields an EM-style CVE algorithm (CV-EM) that converges to the MLE and preserves the theoretical variance. Empirically, CV-EM demonstrates faster convergence and greater numerical stability than standard root-finding methods for the bivariate Normal distribution, and the approach extends to Hutchinson-type trace estimators by enabling MLE-like updates through CV weights. The results offer a unifying, reproducible framework for evaluating and computing estimators that blend MLE and CVE in sketching contexts, with practical heuristics for when CV weights are known.

Abstract

Maximum likelihood estimators (MLE) and control variate estimators (CVE) have been used in conjunction with known information across sketching algorithms and applications in machine learning. We prove that under certain conditions in an exponential family, an optimal CVE will achieve the same asymptotic variance as the MLE, giving an Expectation-Maximization (EM) algorithm for the MLE. Experiments show the EM algorithm is faster and numerically stable compared to other root finding algorithms for the MLE for the bivariate Normal distribution, and we expect this to hold across distributions satisfying these conditions. We show how the EM algorithm leads to reproducibility for algorithms using MLE / CVE, and demonstrate how the EM algorithm leads to finding the MLE when the CV weights are known.
Paper Structure (21 sections, 12 theorems, 91 equations, 15 figures, 3 tables, 1 algorithm)

This paper contains 21 sections, 12 theorems, 91 equations, 15 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Review of Exponential Families
  3. Our Contributions and Paper Organization
  4. Background Information and Setting Up the Problem
  5. Review of Maximum Likelihood Estimation
  6. Review of Control Variates
  7. Equivalence of MLE and CV
  8. Discussion
  9. Experiments
  10. Finding the MLE via CVE weights
  11. Conclusion, Broader Impact, and Future Work
  12. Exponential Family Example with Bivariate Normal
  13. Proofs
  14. Discussion of the EM Algorithm
  15. Discussion of convergence and variance with small $k$
  16. ...and 6 more sections

Key Result

Theorem 1

(Informal Theorem equal_variances_theorem_thingy) Suppose our observations come from an exponential family, with parameters divided into a set $\mathcal{A}$ to be estimated, and $\mathcal{B}$ which is known, and each $\mu_i \equiv \frac{\partial \psi(\vec{\eta})}{\partial \eta_i} = \mathbb E\left(

Figures (15)

  • Figure 1: Idea behind estimation via sketching algorithms. The goal is to use known parameters with MLE or CVE to get a reduced variance estimate. We combine MLE and CVE in Algorithm \ref{['EM_algo_format']} to get an estimator converging to the MLE faster with less numerical error than MLE/CVE alone.
  • Figure 2: The exponential family $\frac{\exp\left\{n \textcolor{red}{\vec{\eta}(} {\nu}_{\mathrm{E}},{\nu}_{\mathrm{K}} \textcolor{red}{)^T}\textcolor{magenta}{\vec{y}} \right\}}{ \exp\left\{ n\textcolor{blue}{\psi(\vec{\eta}(} {\nu}_{\mathrm{E}},{\nu}_{\mathrm{K}} \textcolor{red}{)} \textcolor{blue}{)} \right\}}g(\mathcal{X})$ for $\mathbf{V}^{\mathrm MLE} = \mathbf{V}^{\mathrm CVE}$.
  • Figure 3: MSE plot when $\|\vec{x}_1\|^2 =\|\vec{x}_2\|^2$ with $\theta = \frac{\pi}{12}$.
  • Figure 4: Boxplots of estimated inner products. Blue horizontal line denotes true inner product.
  • Figure 5: Boxplots of update steps till convergence at $k = \{20,40,60,80,100\}$
  • ...and 10 more figures

Theorems & Definitions (22)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 3
  • Theorem 4
  • Corollary 1
  • proof
  • Lemma 1
  • proof
  • Theorem 5
  • ...and 12 more