ERGO-ML: The assembly histories of HSC galaxy images via invertible neural networks, contrastive learning, and cosmological simulations

TL;DR

This work demonstrates that optical galaxy images contain measurable imprints of merger and assembly histories when combined with simulation-based inference. By mapping images to a 256-dimensional representation via NNCLR and conditioning a FrEIA-based cINN on this representation, the authors infer posteriors for ex-situ fractions, last major merger mass, and lookback time, validated against TNG ground truth and transferred to EAGLE mocks. The approach yields robust ex-situ fraction estimates (within ~10% for 80% of cases) and accurate merger masses at higher stellar masses, while lookback time remains challenging; applying to 750k real HSC galaxies reveals consistent trends with mass and morphology but systematically lower ex-situ fractions than in the simulations. Overall, the method establishes a feasible path to extract galaxy assembly histories from images, enabling direct comparisons between observations and multiple cosmological simulations, with clear avenues for improvement through richer contextual data and expanded multi-wavelength information.

Abstract

In this paper of ERGO-ML (Extracting Reality from Galaxy Observables with Machine Learning), we develop a model that infers the merger/assembly histories of galaxies directly from optical images. We apply the self-supervised contrastive learning framework NNCLR (Nearest-Neighbor Contrastive Learning of visual Representations) on realistic HSC mock images (g,r,i - bands) produced from galaxies simulated within the TNG50 and TNG100 flagship runs of the IllustrisTNG project. The resulting representation is then used as conditional input for a cINN (conditional Invertible Neural Network) to gain posteriors for merger/assembly statistics, particularly the lookback time and stellar mass of the last major merger and the fraction of ex-situ stars. Through validation against the ground truth available for simulated galaxies, we assess the performance of our model, achieving good accuracy in inferring the stellar ex-situ fraction ($\le \pm 10$ per cent for 80 per cent of the test sample) and the mass of the last major merger (within $\pm 0.5 \log \MSUN$ for stellar masses $>10^{9.5} \MSUN$ ). We successfully apply the TNG-trained model to simulated mocks from the EAGLE simulation, demonstrating that our model is applicable outside of the TNG domain. We use our simulation-based model to infer aspects of the history of observed galaxies, in particular for HSC images that are close to the domain of TNG ones. We recover the trend of increasing ex-situ stellar fraction with stellar mass and more spherical morphology, but we also identify a discrepancy between TNG and HSC: on average, observed galaxies generally exhibit lower ex-situ fractions. Despite challenges such as information loss (e.g. projection effects and surface brightness limits) and domain shifts (from simulations to observations), our results demonstrate the feasibility of extracting the merger past of galaxies from their optical images.

Figures (16)

• Figure 1: Comparison of the observed HSC dataset with the simulated galaxy images from TNG50, TNG100 and EAGLE. The distributions of all considered datasets are shown in redshift, apparent I-band magnitude, and Petrosian radius (from left to right, see Table \ref{['tab:observable_properties']} for definitions). The HSC galaxies are divided into three subsets: those matched with TNG100, with TNG50 and with EAGLE, due to differences in volume and resolution across the simulations. A larger smoothing kernel is applied to the redshift data, as the TNG simulations are available only for discrete snapshots. Galaxies with an OOD (Out of Domain) score larger than 1.2, indicating high dissimilarity, have been excluded from the analysis compared to Eisert_2024. We find, albeit do not show, that this additional cut does not significantly impact the overall matching between the datasets. The distributions of the unmatched, parent datasets for TNG50 and TNG100 are shown in Figure 2 of Eisert_2024. Note that the redshift distribution for EAGLE is not displayed, as the EAGLE sample used in this work consists of data from only one snapshot/redshift at $z=0.1$. Furthermore we do not display the matched distributions for SIMBA as we use those galaxies only for comparison on image realism but not for testing/validation of our models. Nevertheless, the SIMBA set undergoes the same matching procedure.
• Figure 2: Overview of the galaxy images used in this work. The left panel displays example images from the five datasets used in this work: HSC (observed) and TNG50, TNG100, EAGLE, and SIMBA (simulated). These datasets have all the same selection function in terms of redshift, Petrosian radius, and apparent i-band luminosity. The right panel shows images from the HSC, TNG50, and TNG100 samples after having applied a similarity cut (OOD score less than 1.2). These are the the types of images used for training our ML model. Each column presents 10 images, sorted into evenly spaced bins of increasing stellar mass (ranging from approximately $10^{10} \rm{M}_{\odot}$ to $10^{12} \rm{M}_{\odot}$). The similarity cut in the right panel visually enhances the resemblance between the images, by desire and construction. Meanwhile, diverse morphological structures are present in all considered samples.
• Figure 3: How well do representations of the observed and simulated galaxy images align to each other? We compare the distributions of TNG100, TNG50, EAGLE, and SIMBA images to the observed ones from HSC in the corresponding 2D-UMAP mapping of the 256-dimensional representations obtained by training a ResNet model using contrastive learning simultaneously on all datasets. In the top-left panel, we show kernel density estimation (KDE) density plots of TNG100 images in blue and of the selection function-matched HSC set in light blue. Analogously, we compare the image density distributions of TNG50 vs. its corresponding matched HSC set (top right) and similar visualizations of the UMAPs for EAGLE (bottom left) and SIMBA (bottom right). Contours indicate isodensity lines derived from KDE in the 2D UMAP space. While there is a significant overlap among the first three sets, slight offsets and differences in point density are also evident. For the SIMBA galaxies, the distributions of images diverge most significantly. Namely, TNG100, TNG50 and EAGLE return galaxy images that are, at the population level, more consistent with observed ones from HSC than SIMBA.
• Figure 4: Out-of-Domain (OOD) Scores of the same image representations of Figure \ref{['fig:umaps']}, to quantitatively assess the realism of simulated galaxies in comparison to HSC images. We show the OOD score distributions for the observed HSC and simulated datasets (TNG50, TNG100, EAGLE, and SIMBA), following the methodology introduced by Eisert_2024 and using the self-supervised contrastive-learning model of Figure \ref{['fig:umaps']}. The OOD score is evaluated across multiple scenarios, including comparisons between pairs of datasets and random splits within each dataset, to understand the inherent scatter: the distribution of self-distances for HSC images is provided as reference and is similar to those of the self-distances across the other simulated samples (not shown). The measurement distinguishes between the "sides" of the comparison, where, for example, TNG100-HSC refers to the distances of TNG100 galaxies relative to HSC, while HSC-TNG100 reflects the reverse. Images with high OOD values denote galaxies that do not resemble well observed ones. From the shape of the distributions, we can see that TNG50 and TNG100 return galaxy samples that are overall more realistic, i.e. more aligned, with HSC galaxies than EAGLE and, to a much larger degree, SIMBA. Namely, there are relatively fewer TNG50 and TNG100 galaxies than in SIMBA that appear inconsistent with HSC data.
• Figure 5: Posterior distributions of the three assembly history statistics inferred in this work with our SBI model and shown for 15 randomly selected galaxies from the TNG100 simulation test sets. Each row represents a single galaxy, with its redshift and unique Subfind ID listed on the left, while each column displays the inferred unobservable statistics of its assembly and merger history. The distributions are depicted for the entire test galaxy sample, i.e. the prior (in grey), our cINN model predictions (in blue), Maximum A-Posteriori (MAP) estimates (in yellow), and the ground truth from the TNG simulations (in red). These distributions are normalized so that the maximum value equals 1. Additionally, we include an extra bin for the last major merger quantities, indicating the fraction of posterior samples falling outside the valid regime, representing the probability that the galaxy has not experienced a major merger in its history. The MAP estimate for this bin is utilized only if the fraction of posterior samples exceeds 50 per cent, indicating a greater than 50 per cent likelihood that no major merger occurred during the lifetime of the respective galaxy.