TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression

TL;DR

This paper tackles sub-optimal covariance estimation in deep heteroscedastic regression by introducing Taylor Induced Covariance (TIC), a closed-form covariance approximation that ties the randomness of predictions to the local gradient and curvature via the Jacobian and Hessian of the mean predictor, plus an independent noise term. It also introduces Task Agnostic Correlations (TAC), a conditioning-based metric that directly assesses the quality of learned correlations in the covariance without ground-truth covariances. The authors show that TIC–learned covariances improve convergence of the negative log-likelihood and provide more faithful uncertainty representations across synthetic data, UCI regression, and 2D human pose estimation, with TAC offering a robust evaluation of covariance quality. The work includes practical guidance on limitations (e.g., Hessian computation) and demonstrates reproducibility via public code.

Abstract

Deep heteroscedastic regression involves jointly optimizing the mean and covariance of the predicted distribution using the negative log-likelihood. However, recent works show that this may result in sub-optimal convergence due to the challenges associated with covariance estimation. While the literature addresses this by proposing alternate formulations to mitigate the impact of the predicted covariance, we focus on improving the predicted covariance itself. We study two questions: (1) Does the predicted covariance truly capture the randomness of the predicted mean? (2) In the absence of supervision, how can we quantify the accuracy of covariance estimation? We address (1) with a Taylor Induced Covariance (TIC), which captures the randomness of the predicted mean by incorporating its gradient and curvature through the second order Taylor polynomial. Furthermore, we tackle (2) by introducing a Task Agnostic Correlations (TAC) metric, which combines the notion of correlations and absolute error to evaluate the covariance. We evaluate TIC-TAC across multiple experiments spanning synthetic and real-world datasets. Our results show that not only does TIC accurately learn the covariance, it additionally facilitates an improved convergence of the negative log-likelihood. Our code is available at https://github.com/vita-epfl/TIC-TAC

Figures (7)

• Figure 1: Motivation. (Left) We learn a varying amplitude sinusoidal with heteroscedastic variance (shaded region). We observe sub-optimal convergence since the predicted variance may be arbitrary and incorrectly minimizes the likelihood. We address this through a Taylor Induced Covariance by tying the randomness of the prediction to its gradient and curvature. (Right) The gradient and curvature quantify the variation in the prediction within a small neighborhood of the input.
• Figure 2: Task Agnostic Correlations (TAC). We propose the TAC metric for covariance evaluation. Given the ground truth ${\bm{y}}$, predicted mean $\hat{{\bm{y}}}$ and covariance ${\bm{\Sigma}}$, TAC quantifies the improvement in the predicted mean given partial observations of the ground truth. TAC uses conditioning of the normal distribution to directly assess the covariance.
• Figure 3: Univariate. We perform experiments on three different sinusoidals, showing that incorporating the gradient and curvature of the predicted mean results in accurate variance estimation. The TIC parameterization also results in an improved convergence of the negative log-likelihood.
• Figure 4: Multivariate Schematic. We present a simple method to simulate heteroscedastic data. We first randomly sample the input ${\bm{x}}$, which in turn is used to sample an initial target ${\bm{y}}$. We then add sample-dependent noise ${\bm{z}}$, giving us the target ${\bm{q}}$, which the network is required to learn.
• Figure 5: Multivariate Results. We plot the Task Agnostic Correlations (TAC) metric mean and standard deviation for all methods from dimensions 4 to 20. The gap between TIC and the baselines widens as the dimensionality increases.