Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

T-Rex: Fitting a Robust Factor Model via Expectation-Maximization

Daniel Cederberg

TL;DR

This work tackles robust covariance estimation in high dimensions by fitting a statistical factor model $\Sigma = FF^T + D$ under elliptical-distribution assumptions. It derives T-Rex, an EM algorithm that integrates Tyler's M-estimator with a latent-variable model to impose the low-rank plus diagonal structure, and shows that Tyler's fixed-point iteration is a special EM case. The method delivers strong robustness to heavy tails and outliers, scales to large $n$, and achieves favorable performance in DOA estimation and robust subspace recovery compared to Gaussian and $t$-based alternatives. Empirical results on synthetic data, DOA benchmarks, and face-subspace tasks demonstrate T-Rex's robustness, efficiency, and practical impact for robust high-dimensional statistics in signal processing and computer vision.

Abstract

Over the past decades, there has been a surge of interest in studying low-dimensional structures within high-dimensional data. Statistical factor models $-$ i.e., low-rank plus diagonal covariance structures $-$ offer a powerful framework for modeling such structures. However, traditional methods for fitting statistical factor models, such as principal component analysis (PCA) or maximum likelihood estimation assuming the data is Gaussian, are highly sensitive to heavy tails and outliers in the observed data. In this paper, we propose a novel expectation-maximization (EM) algorithm for robustly fitting statistical factor models. Our approach is based on Tyler's M-estimator of the scatter matrix for an elliptical distribution, and consists of solving Tyler's maximum likelihood estimation problem while imposing a structural constraint that enforces the low-rank plus diagonal covariance structure. We present numerical experiments on both synthetic and real examples, demonstrating the robustness of our method for direction-of-arrival estimation in nonuniform noise and subspace recovery.

T-Rex: Fitting a Robust Factor Model via Expectation-Maximization

TL;DR

This work tackles robust covariance estimation in high dimensions by fitting a statistical factor model under elliptical-distribution assumptions. It derives T-Rex, an EM algorithm that integrates Tyler's M-estimator with a latent-variable model to impose the low-rank plus diagonal structure, and shows that Tyler's fixed-point iteration is a special EM case. The method delivers strong robustness to heavy tails and outliers, scales to large , and achieves favorable performance in DOA estimation and robust subspace recovery compared to Gaussian and -based alternatives. Empirical results on synthetic data, DOA benchmarks, and face-subspace tasks demonstrate T-Rex's robustness, efficiency, and practical impact for robust high-dimensional statistics in signal processing and computer vision.

Abstract

Over the past decades, there has been a surge of interest in studying low-dimensional structures within high-dimensional data. Statistical factor models i.e., low-rank plus diagonal covariance structures offer a powerful framework for modeling such structures. However, traditional methods for fitting statistical factor models, such as principal component analysis (PCA) or maximum likelihood estimation assuming the data is Gaussian, are highly sensitive to heavy tails and outliers in the observed data. In this paper, we propose a novel expectation-maximization (EM) algorithm for robustly fitting statistical factor models. Our approach is based on Tyler's M-estimator of the scatter matrix for an elliptical distribution, and consists of solving Tyler's maximum likelihood estimation problem while imposing a structural constraint that enforces the low-rank plus diagonal covariance structure. We present numerical experiments on both synthetic and real examples, demonstrating the robustness of our method for direction-of-arrival estimation in nonuniform noise and subspace recovery.
Paper Structure (36 sections, 2 theorems, 38 equations, 10 figures)

This paper contains 36 sections, 2 theorems, 38 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Background on Tyler's estimator
  4. Background on expectation-maximization
  5. An EM algorithm for fitting a robust factor model
  6. The latent variable model
  7. The expectation and maximization steps
  8. The complete algorithm
  9. Interpretation of T-Rex.
  10. Interpretation of Tyler's fixed-point iteration as EM.
  11. Implementation
  12. Initialization and termination.
  13. Evaluating the M-step.
  14. Exploiting low-rank structure.
  15. Iteration cost.
  16. ...and 21 more sections

Key Result

Lemma 1

Assume the joint density of $(X, R)$ is given by e:latent-variable-model. Then the conditional distribution of $R$ given $X = x$ is the same as the distribution of $(1/\sqrt{x^T \Sigma_{FD}^{-1} x}) S$ where $S \sim \chi(n)$.

Figures (10)

  • Figure 1: The MSE \ref{['e:MSE-synthetic']} on three different types of data. The error bars represent one standard deviation above and below the average error over 100 runs.
  • Figure 2: Left. The runtime of T-Rex and Tyler versus the dimension $n$ of the covariance. We ran both methods for a fixed number of 20 iterations. Right. The MSE of T-Rex and Tyler.
  • Figure 3: Left. The MSE versus the number of snapshots in the Gaussian setting. Right. The MSE versus the number of snapshots in the heavy-tailed setting.
  • Figure 4: Pseudospectrum for T-Rex, G-FA, SC and STE for $m = 100$ snapshots in the heavy-tailed scenario.
  • Figure 5: Face images projected onto nine-dimensional linear models. The original images (leftmost column) are projected onto subspaces that were fitted using five different methods. The first two rows show two in-sample faces. The last two rows show two out-of-sample faces. (The out-of-sample images were not used to fit the linear subspaces.)
  • ...and 5 more figures

Theorems & Definitions (2)

  • Lemma 1
  • Proposition 1