Maximum likelihood inference for high-dimensional problems with multiaffine variable relations

TL;DR

This paper proposes a novel Alternating and Iteratively-Reweighted Least Squares (AIRLS) algorithm, and proves its convergence for problems with Generalized Normal Distributions, and provides an efficient method to compute the variance of the estimates obtained using AIRLS.

Abstract

Maximum Likelihood Estimation of continuous variable models can be very challenging in high dimensions, due to potentially complex probability distributions. The existence of multiple interdependencies among variables can make it very difficult to establish convergence guarantees. This leads to a wide use of brute-force methods, such as grid searching and Monte-Carlo sampling and, when applicable, complex and problem-specific algorithms. In this paper, we consider inference problems where the variables are related by multiaffine expressions. We propose a novel Alternating and Iteratively-Reweighted Least Squares (AIRLS) algorithm, and prove its convergence for problems with Generalized Normal Distributions. We also provide an efficient method to compute the variance of the estimates obtained using AIRLS. Finally, we show how the method can be applied to graphical statistical models. We perform numerical experiments on several inference problems, showing significantly better performance than state-of-the-art approaches in terms of scalability, robustness to noise, and convergence speed due to an empirically observed super-linear convergence rate.

Figures (9)

• Figure 1: Relative Frobenius error of the parameter estimates of a double-integrator system using 50 thousand samples for subspace identification subspace_compa, AIRLS, Recursive Total Least Squares (RTLS) rtls_compa, and the Extended Kalman Filter (EKF) ekf_compa. The data has a proportion of outliers up to 5%. The vertical axis is in log scale.
• Figure 2: Relative estimation error of power grid parameters using Ordinary Least Squares (OLS), Least Absolute Shrinkage and Selection Operator (Lasso), MLE, and MAP.
• Figure 3: Example of a supply demand Bayesian network model with taxes or subsidies.
• Figure 4: Comparison of convergence speed of four algorithms for the inference of $P_t$ in the example \ref{['eq_prob_econ_def']} with unknown $\tau$, $T=2$, and $n_T = 1$.
• Figure 5: Comparison of ZOGD and AIRLS for scaling with $T$, and for robustness to noise. The subfigures show (a) the computation time as a function of $T$ and (b) the RRMS error of the estimate of $P_t$ and $\tau$ depending on the average noise in $S_t$ and $D_t$.