Combining datasets with different ground truths using Low-Rank Adaptation to generalize image-based CNN models for photometric redshift prediction

TL;DR

This work tackles the challenge of leveraging heterogeneous ground-truth sources for galaxy redshift regression by using Low-Rank Adaptation (LoRA) to finely adjust a CNN trained on photometric redshifts with spectroscopic fine-tuning data. The authors compare LoRA to traditional transfer learning and full retraining across three datasets (TransferZ-Images, GalaxiesML, and the combined Combo), demonstrating that LoRA reduces bias and scatter relative to TL and requires far less computation than retraining on all data. They show that approximately 40–50% of the spectroscopic data during LoRA fine-tuning can yield near-Combo performance, highlighting LoRA as a practical middle ground for data-sparse regimes. The study suggests LoRA’s potential to adapt pre-trained astrophysical models to multiple new datasets, a valuable capability for upcoming surveys like LSST that will demand efficient leveraging of existing models. <math>LoRA</math> thus emerges as a promising tool for precision cosmology when ground-truth data are unevenly distributed.

Abstract

In this work, we demonstrate how Low-Rank Adaptation (LoRA) can be used to combine different galaxy imaging datasets to improve redshift estimation with CNN models for cosmology. LoRA is an established technique for large language models that adds adapter networks to adjust model weights and biases to efficiently fine-tune large base models without retraining. We train a base model using a photometric redshift ground truth dataset, which contains broad galaxy types but is less accurate. We then fine-tune using LoRA on a spectroscopic redshift ground truth dataset. These redshifts are more accurate but limited to bright galaxies and take orders of magnitude more time to obtain, so are less available for large surveys. Ideally, the combination of the two datasets would yield more accurate models that generalize well. The LoRA model performs better than a traditional transfer learning method, with $\sim2.5\times$ less bias and $\sim$2.2$\times$ less scatter. Retraining the model on a combined dataset yields a model that generalizes better than LoRA but at a cost of greater computation time. Our work shows that LoRA is useful for fine-tuning regression models in astrophysics by providing a middle ground between full retraining and no retraining. LoRA shows potential in allowing us to leverage existing pretrained astrophysical models, especially for data sparse tasks.

Figures (7)

• Figure 1: The redshift number distributions of the 557 common galaxies using spectroscopic ground truth from GalaxiesML (blue) and photometric ground truth from TranferZ-Images (red).
• Figure 2: Visualization of LoRA implementation on a ResNet + regressor Model, based on Fig 1. in hu2021.
• Figure 3: Bias (left), scatter (middle), and catastrophic outlier rate performance metrics (right) for all four models. The metrics are evaluated on all three datasets (TransferZ-Images (red), GalaxiesML (blue), Combo (purple)). The green band for Bias and Scatter are the LSST Science Requirements. The black stars represent the best (lowest) metric for each dataset. Upward black arrows indicates the bar overflows the plot. Low-Rank adaptation (CNN-LoRA) performs better than traditional transfer learning (CNN-TL), but not as well as retraining on the entire dataset (CNN-Combo).
• Figure 4: Bias (left), scatter (middle) and catastrophic outlier rate (right) as a function of fraction of GalaxiesML used to fine-tune using CNN-LoRA. The green band for Bias and Scatter are the LSST Science Requirements. Using just 10$\%$ of the data, the model's scatter on GalaxiesML improves and on TransferZ reduces drastically. The performance of CNN-LoRA plateaus when 40-50$\%$ of GalaxiesML is used for training; only a fraction of the dataset is needed for the best fine-tuning.
• Figure 5: The true redshift vs predicted redshift for four models (a) CNN-Base, (b) CNN-TL, (c) CNN-LoRA and (d) CNN-Combo, evaluated on the Combo dataset. The color bar describes the log norm of the number of galaxies in each 2D histogram bin. The identity line (1-1 line) represents perfect agreement between ground truth and predictions. Low-Rank adaptation (CNN-LoRA) performs better than traditional transfer learning (CNN-TL), but not as well as retraining on the entire dataset (CNN-Combo).