An Unbiased Variance Estimator with Denominator $N$
Dai Akita
TL;DR
This paper asks whether an unbiased variance estimator can be achieved with a fixed denominator $N$ by adjusting the mean estimator rather than the denominator itself. The central idea, average-adjusted unbiased variance (AAUV), identifies linear mean estimators $\hat{X}=\sum c_n X_n$ that satisfy $\sum c_n=1$ and $\sum c_n^2=\frac{2}{N}$, yielding $\hat{s}^2=\frac{1}{N}\sum (X_n-\hat{X})^2$ with $\mathbb{E}[\hat{s}^2]=\sigma^2$; the half-sample mean is a canonical example. The paper also develops a continuum of unbiased estimators by interpolating between AAUV and the standard mean with $\tilde{X}_\lambda=\lambda\hat{X}+(1-\lambda)\bar{X}$, giving $s_\lambda^2=\frac{1}{N-1+\lambda^2}\sum (X_n-\tilde{X}_\lambda)^2$ and enabling denominators $K=N-1+\lambda^2$; symmetrization over permutations recovers the classical $s^2$. While AAUVs illuminate a broader design space for unbiased variance estimation, they generally incur higher variance and are non-symmetric, limiting practical gains; the paper briefly discusses extending the approach to third and higher-order central moments, noting partial constructive results and open questions. Overall, the work clarifies how fixed-denominator variance estimation can be achieved via mean adjustment and highlights a path toward continuous interpolation with the standard estimator, along with practical and theoretical caveats.
Abstract
Standard practice obtains an unbiased variance estimator by dividing by $N-1$ rather than $N$. Yet if only half the data are used to compute the mean, dividing by $N$ can still yield an unbiased estimator. We show that an alternative mean estimator $\hat{X} = \sum c_n X_n$ can produce such an unbiased variance estimator with denominator $N$. These average-adjusted unbiased variance (AAUV) permit infinitely many unbiased forms, though each has larger variance than the usual sample variance. Moreover, permuting and symmetrizing any AAUV recovers the classical formula with denominator $N-1$. We further demonstrate a continuum of unbiased variances by interpolating between the standard and AAUV-based means. Extending this average-adjusting method to higher-order moments remains a topic for future work.