Why Machine Learning Models Systematically Underestimate Extreme Values

TL;DR

This work reveals a fundamental attenuation bias in astronomical regression tasks, showing that input measurement errors systematically shrink regression coefficients and predicted labels, independent of training size or label accuracy. The authors derive analytic expressions for univariate and multivariate cases, highlighting how signal-to-noise and feature correlations modulate bias, and they validate these results with extensive simulations and spectral-emulation experiments using APOGEE-like data. They demonstrate that while correlated features can mitigate bias in idealized scenarios, real spectra—especially at lower resolutions or SNR—still exhibit percent-level biases with practical implications for stellar parameters, abundances, and galactic inferences. The paper argues for caution with discriminative mappings from noisy observables and suggests generative forward-modeling and empirical calibration as robust strategies to address attenuation bias in astronomical data analysis.

Abstract

A persistent challenge in astronomical machine learning is a systematic bias where predictions compress the dynamic range of true values-high values are consistently predicted too low while low values are predicted too high. Understanding this bias has important consequences for astronomical measurements and our understanding of physical processes in astronomical inference. Through analytical examination of linear regression, we show that this bias arises naturally from measurement uncertainties in input features and persists regardless of training sample size, label accuracy, or parameter distribution. In the univariate case, we demonstrate that attenuation becomes important when the ratio of intrinsic signal range to measurement uncertainty ($σ_{\text{range}}/σ_x$) is below $O(10)$-a regime common in astronomy. We further extend the theoretical framework to multivariate linear regression and demonstrate its implications using stellar spectroscopy as a case study. Even under optimal conditions-high-resolution APOGEE-like spectra ($R=24,000$) with high signal-to-noise ratios (SNR=100) and multiple correlated features-we find percent-level bias. The effect becomes even more severe for modern-day low-resolution surveys like LAMOST and DESI due to the lower SNR and resolution. These findings have broad implications, providing a theoretical framework for understanding and addressing this limitation in astronomical data analysis with machine learning.

Figures (11)

• Figure 1: Visualization of attenuation bias through density plots comparing predicted versus true values. Left panel shows predictions with $\sigma_{\text{range}}/\sigma_x = 10$, where the fitted regression line between the predicted $y$ as a function of true $y$ (red solid, $\lambda_y = 0.99$) closely follows the perfect prediction line (black dashed) with only 1% bias. Right panel shows the case where measurement uncertainties are comparable to signal variance ($\sigma_{\text{range}}/\sigma_x = 1$), where the fitted line ($\lambda_y = 0.5$).
• Figure 2: Impact of measurement uncertainties on coefficient attenuation bias in univariate regression. The attenuation factor for the regression coefficient ($\lambda_\beta$) as a function of $\sigma_{\text{range}}/\sigma_x$ shows analytical predictions (lines) and numerical simulations (points) for linear (blue-solid), quadratic (orange-dashed), and cubic (green-dotted) terms. For each polynomial order, $\lambda_\beta$ is defined as the attenuation bias factor of the highest order coefficient.
• Figure 3: Effect of dimensionality on attenuation bias in coefficients and predictions for independently drawn input features in multivariate linear regression. Left panel shows the distribution of coefficient attenuation factors ($\lambda_{\beta} = \hat{\beta}/\beta_{\text{true}}$), pooling results across all coefficients and simulations for each $p$, with boxcars showing the distribution over all coefficients and 100 simulations. Right panel shows prediction attenuation factors ($\lambda_y$), obtained by regressing predicted values against true values across all samples in each simulation, with boxcars showing the distribution over all simulations. For both panels, we examined three different signal-variance-to-noise ratios ($\sigma_{\text{range}}/\sigma_x = 2, 3, 10$). The boxes indicate the interquartile range (IQR) with the median shown as a horizontal line, and whiskers extend to points within 1.5×IQR. The open circles show individual realizations that fall beyond the whiskers. The dashed lines show the theoretical predictions of $\lambda = 1/(1 + (\sigma_x/\sigma_{\text{range}})^2)$ for each signal-variance-to-noise ratio. The inset plots show the variance of the coefficient estimates (left) and predicted values (right) as a function of dimensionality, with points showing empirical results and dashed lines showing theoretical predictions.
• Figure 4: Prediction attenuations versus training sample size for different ratios of signal range to measurement error with fixed input dimension $p=5$ for multivariate linear regression with independent input features. Solid lines show mean attenuation factors ($\lambda_y$) with shaded regions indicating $\pm 1$ standard deviation calculated from 100 independent simulations. Results shown for three ratios ($\sigma_{\text{range}}/\sigma_x = 2, 3, 5$). For each ratio, horizontal dashed line indicates theoretical attenuation limit $\lambda = 1/(1 + (\sigma_x/\sigma_{\text{range}})^2)$, curved dashed lines show $1/\sqrt{n}$ convergence scaling from coefficient estimate variance. Solid black line at $\lambda_y = 1$ indicates case of no attenuation.
• Figure 5: Effect of dimensionality on attenuation bias for perfectly correlated features. Left panel shows coefficient attenuation factors ($\lambda_{\beta}$) and right panel shows prediction attenuation factors ($\lambda_y$) for different numbers of perfectly correlated input dimensions. Results are shown for three ratios of signal range to measurement error ($\sigma_{\text{range}}/\sigma_x = 2, 3, 10$) based on 100 independent simulations. Features are generated as linear transformations of a uniform random variable on [0,1] ($\sigma_{\text{range}} = 1/\sqrt{12}$) with scaling factors $a_j$ varying linearly from 0.2 to 0.4 across dimensions, chosen to ensure labels remain of order unity. In the left panel, box plots show the distribution of coefficient attenuation across all coefficients and simulations, with boxes offset horizontally for visibility across different $\sigma_{\text{range}}/\sigma_x$ ratios. The right panel shows the mean prediction attenuation with $\pm 1$ standard deviation across simulations. Dashed lines in both panels show theoretical predictions. The solid black line at $\lambda = 1$ indicates no attenuation.