Diagnosing the Effects of Spectroscopic Training Set Imperfection on Photometric Redshift Performance

TL;DR

This work investigates how imperfections in spectroscopic training sets affect photometric redshift performance for LSST-scale surveys using a RAIl-based pipeline and multiple estimators. By generating ground-truth posteriors with a normalizing-flow model and applying inverse redshift incompleteness, survey-based degradations, and LSST-like noise, the study evaluates various metrics (notably Kullback–Leibler Divergence, Wasserstein distance, and PIT) to diagnose covariate-shift effects. The results show that non-representativeness of training data has a stronger impact than incompleteness alone, and that no single metric suffices to characterize performance; a combination of distribution-to-distribution, CDF-based, and calibration metrics provides a fuller picture. The findings guide metric selection for vetting photo-$z$ estimators and highlight the practical importance of representative training samples for reliable redshift recovery in large surveys.

Abstract

Most LSST extragalactic science will rely on photometric redshifts (photo-$z$) to extract distance information for the galaxies. However, an incomplete or non-representative training set can introduce bias into photo-$z$ estimation. It is necessary to understand how various forms of training set imperfection, such as incompleteness and non-trivial spectroscopic target selection, affect photo-$z$ estimation algorithms, and to identify metrics best-suited to quantify the impact. This work aims to systematically study metrics for diagnosing how various photo-$z$ methods react to certain types of training set incompleteness and non-representativeness. We use methods available through the open-source Python library Redshift Assessment Infrastructure Layers (RAIL) to systematically test the algorithms CMNN, GPz, FlexZBoost, and PZFlow on mock training data degraded in accordance with several existing spectroscopic sky surveys, as well as under conditions of inverse redshift incompleteness, which approximately mimics observed patterns of incompleteness at high redshift. We employ the algorithm TrainZ as a control. Finally, we quantify photo-$z$ algorithm performance using a variety of statistical metrics implemented externally to RAIL. We determine that the Kullback-Liebler Divergence, Wasserstein Distance, and Probability Integral Transform are particularly informative metrics with which to assess the impact of training set imperfection on algorithmic performance. We also find that inverse redshift incompleteness effects alone lack the complexity to realistically represent anticipated training data.

Figures (11)

• Figure 1: A flowchart depicting the pipeline structure used to perform creation, degradation, and estimation. Boxes with rounded corners are RAIL stages, while boxes with sharp corners are data products. Creation stages are shown in blue, degradation stages in peach, the informer stage that trains a given photo-$z$ estimation method in yellow, and the estimator stage in green. The sections where more information may be found for the RAIL stages used are indicated in the legend, and Sec. \ref{['sec:overview']} describes the overall workflow in detail.
• Figure 2: Regions in redshift-magnitude space remaining populated after inverse redshift incompleteness degradation, performed with several different values of $z_0$ (as labeled at the top of each column). The colormap indicates the density of surviving data, while the pink points have no inverse redshift incompleteness degradation and are our test set. The top 6 rows show the 6 LSST passbands, while the bottom row shows the 1D histograms of redshift before and after the degradation.
• Figure 3: Regions in redshift-magnitude space remaining populated after spectroscopic survey degradation. The colormap indicates the density of surviving data, while the pink data have received no spectroscopic survey degradation and are our test set (see Fig. \ref{['fig:pipeline']}). Note the variation in degrees of test set coverage and localization of data in redshift-magnitude space across different surveys. The top 6 rows show the 6 LSST passbands, while the bottom row shows the 1D histograms of redshift before and after the degradation.
• Figure 4: Bars correspond to approximate percentages of our test set covered by each spectroscopic training set, shown for each LSST band. The range of redshift-magnitude space captured in each band was covered with a discretized grid, with bins of width 0.1 in the redshift direction and 1 in the magnitude direction. The reported percentage is the percentage of the squares containing at least one test set galaxy that also contain at least one training set galaxy. Dashed lines correspond to the average coverage across all six bands for each spectroscopic degrader.
• Figure 5: Mode point estimates obtained from estimator outputs under representative training conditions, versus true spectroscopic galaxy redshifts. Clustering along the diagonal is ideal case, which is shown by the dashed orange line. Red dashed lines show the 3$\sigma$ distance, calculated via outlier rejection. TrainZ is not shown, as the plot is not illustrative: it assigns every galaxy the same distribution, thus all modes are the same. $>3\sigma$ outlier rates are shown in the upper left corner of each panel.