Why Machine Learning Models Systematically Underestimate Extreme Values II: How to Fix It with LatentNN

TL;DR

Attenuation bias causes neural networks to underpredict extreme values when inputs carry measurement errors, a problem particularly acute in astronomy where $\lambda_\beta = 1/(1 + (\sigma_x/\sigma_{\rm range})^2)$. LatentNN generalizes the errors-in-variables approach by introducing latent inputs $x_{\rm latent}$ and jointly optimizing them with network parameters $\boldsymbol{\theta}$ under a joint Gaussian likelihood, extending Deming regression to nonlinear function approximation. Across one-dimensional and multivariate synthetic tests, and in a stellar-spectra application, LatentNN consistently reduces attenuation bias (achieving $\lambda_y \approx 1$ for $\text{SNR}_x \gtrsim 2$ in many settings) while maintaining generalization, with performance depending on the number and structure of informative features. The method frames attenuation bias within a hierarchical Bayesian perspective (MAP of latent inputs), offers generalizations to heteroscedastic and non-Gaussian noise, and provides practical implications for spectroscopic surveys by yielding more accurate stellar parameters and abundances in low-SNR data; code is publicly available at the LatentNN repository.

Abstract

Attenuation bias -- the systematic underestimation of regression coefficients due to measurement errors in input variables -- affects astronomical data-driven models. For linear regression, this problem was solved by treating the true input values as latent variables to be estimated alongside model parameters. In this paper, we show that neural networks suffer from the same attenuation bias and that the latent variable solution generalizes directly to neural networks. We introduce LatentNN, a method that jointly optimizes network parameters and latent input values by maximizing the joint likelihood of observing both inputs and outputs. We demonstrate the correction on one-dimensional regression, multivariate inputs with correlated features, and stellar spectroscopy applications. LatentNN reduces attenuation bias across a range of signal-to-noise ratios where standard neural networks show large bias. This provides a framework for improved neural network inference in the low signal-to-noise regime characteristic of astronomical data. This bias correction is most effective when measurement errors are less than roughly half the intrinsic data range; in the regime of very low signal-to-noise and few informative features. Code is available at https://github.com/tingyuansen/LatentNN.

Figures (7)

• Figure 1: Schematic illustration of attenuation bias. Blue circles show true positions $(x_{\rm true}, y)$ following $y = 2x$ (black line); red squares show observed positions $(x_{\rm obs}, y)$ after adding measurement error with $\mathrm{SNR}_x = 1$. Grey arrows indicate horizontal displacement from true to observed $x$ positions. The tilted grey shaded bands show the $\pm 1\sigma_x$ (darker) and $\pm 2\sigma_x$ (lighter) uncertainty regions around the true relationship. A linear fit to the observed data (red dashed line) yields a shallower slope, demonstrating how input measurement errors systematically bias regression coefficients toward zero.
• Figure 2: Attenuation bias in neural networks for the true function $y = 2x$. Top: Predicted versus true $y$ on held-out test data for an MLP (2 hidden layers, 64 units) at three $\mathrm{SNR}_x$ values. At $\mathrm{SNR}_x = 0.3$, predictions collapse to $\lambda_y \approx 0.1$; at $\mathrm{SNR}_x = 1$, $\lambda_y \approx 0.5$; at $\mathrm{SNR}_x = 3$, $\lambda_y \approx 0.9$. Bottom: Attenuation factor $\lambda_y$ versus $\mathrm{SNR}_x$ for linear regression (blue open circles) and neural network (green filled squares). Both follow the theoretical curve $\lambda_y = 1/(1 + \mathrm{SNR}_x^{-2})$ (black solid line), demonstrating that model complexity does not protect against attenuation bias.
• Figure 3: LatentNN demonstration at $\mathrm{SNR}_x=1$. Top: Predicted versus true $y$ on the test set for standard neural networks with the MLP arhitecture (left, $\lambda_y \approx 0.5$) and LatentNN (right, $\lambda_y \approx 1$). LatentNN corrects the attenuation bias. Bottom left: Learned functions. The standard MLP (orange) learns an attenuated slope, while LatentNN (green) recovers the true function $f(x) = 2x$ (black dashed). The gray shaded region indicates the noiseless training x-value range. Bottom right: LatentNN training losses. The prediction loss decreases while the $x_{\rm latent}$ likelihood increases, reflecting the trade-off as latent values shift from noisy observations toward true values.
• Figure 4: Attenuation factor $\lambda_y$ versus $\mathrm{SNR}_x$ for three $\mathrm{SNR}_y$ levels (3, 10, 30; different colors). Error bars show standard deviation over 8 runs with different random seeds, using a three-fold data split (train/validation/test) to avoid data leakage in hyperparameter selection. Standard MLP (open circles, dotted) follows the theoretical attenuation curve (black solid) regardless of $\mathrm{SNR}_y$. LatentNN (filled squares, solid) maintains mean $\lambda_y \approx 1$ across all tested SNR values. Even at $\mathrm{SNR}_x = 1$, LatentNN achieves unbiased predictions on average, though with increased run-to-run scatter at lower $\mathrm{SNR}_x$ as the problem becomes increasingly ill-conditioned.
• Figure 5: Attenuation factor $\lambda_y$ versus $\mathrm{SNR}_x$ for correlated multivariate inputs with $p = 3, 10, 30$ dimensions (panels). Error bars show standard deviation over 8 runs. Black dashed line marks $\lambda_y = 1$ (no attenuation). Gray solid curve shows the analytic prediction for linear regression from Paper I; standard MLP (open circles, dotted) closely follows this theoretical curve, confirming that model complexity does not mitigate the bias. With increasing $p$, attenuation is mitigated because independent noise in different dimensions must conspire coherently to displace observations along the signal direction. LatentNN (filled squares, solid) maintains $\lambda_y \approx 1$ for $p \geq 10$ at $\mathrm{SNR}_x \geq 2$, with mild degradation ($\lambda_y \approx 0.9$) at $\mathrm{SNR}_x = 1$. For $p=3$, LatentNN correction degrades at low SNR ($\lambda_y \approx 0.7$ at $\mathrm{SNR}_x = 1$), but still outperforms the standard MLP ($\lambda_y \approx 0.2$).