Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Masked Autoencoders for Learning Image-Spectrum Associations for Galaxy Evolution and Cosmology

Morgan Himes, Samiksha Krishnamurthy, Andrew Lizarraga, Srinath Saikrishnan, Vikram Seenivasan, Jonathan Soriano, Ying Nian Wu, Tuan Do

TL;DR

This work tackles the challenge of learning cross-modal representations from galaxy images and spectra when spectra are sparse by training a Multi-Modal Masked Autoencoder (MMAE) on a large, jointly labeled dataset of HSC PDR2 images and DESI DR1 spectra up to $z\sim4$. The model uses patch-based tokenization, cross-attention fusion, and integrates a redshift regression head directly into masked autoencoding, achieving competitive redshift scatter compared to prior multi-modal methods while reconstructing broad spectral continua and galaxy morphologies. However, it struggles with fine-detail morphology and weaker emission lines, and its high-redshift performance is limited by data availability, motivating physics-aware losses and the inclusion of additional modalities for robust foundation-model-style learning in astronomy. Overall, the study demonstrates both the potential and current limitations of masked autoencoders for multi-modal astrophysical data and lays groundwork for extending to textual metadata and beyond.

Abstract

Upcoming surveys will produce billions of galaxy images but comparatively few spectra, motivating models that learn cross-modal representations. We build a dataset of 134,533 galaxy images (HSC-PDR2) and spectra (DESI-DR1) and adapt a Multi-Modal Masked Autoencoder (MMAE) to embed both images and spectra in a shared representation. The MMAE is a transformer-based architecture, which we train by masking 75% of the data and reconstructing missing image and spectral tokens. We use this model to test three applications: spectral and image reconstruction from heavily masked data and redshift regression from images alone. It recovers key physical features, such as galaxy shapes, atomic emission line peaks, and broad continuum slopes, though it struggles with fine image details and line strengths. For redshift regression, the MMAE performs comparably or better than prior multi-modal models in terms of prediction scatter even when missing spectra in testing. These results highlight both the potential and limitations of masked autoencoders in astrophysics and motivate extensions to additional modalities, such as text, for foundation models.

Multi-Modal Masked Autoencoders for Learning Image-Spectrum Associations for Galaxy Evolution and Cosmology

TL;DR

This work tackles the challenge of learning cross-modal representations from galaxy images and spectra when spectra are sparse by training a Multi-Modal Masked Autoencoder (MMAE) on a large, jointly labeled dataset of HSC PDR2 images and DESI DR1 spectra up to . The model uses patch-based tokenization, cross-attention fusion, and integrates a redshift regression head directly into masked autoencoding, achieving competitive redshift scatter compared to prior multi-modal methods while reconstructing broad spectral continua and galaxy morphologies. However, it struggles with fine-detail morphology and weaker emission lines, and its high-redshift performance is limited by data availability, motivating physics-aware losses and the inclusion of additional modalities for robust foundation-model-style learning in astronomy. Overall, the study demonstrates both the potential and current limitations of masked autoencoders for multi-modal astrophysical data and lays groundwork for extending to textual metadata and beyond.

Abstract

Upcoming surveys will produce billions of galaxy images but comparatively few spectra, motivating models that learn cross-modal representations. We build a dataset of 134,533 galaxy images (HSC-PDR2) and spectra (DESI-DR1) and adapt a Multi-Modal Masked Autoencoder (MMAE) to embed both images and spectra in a shared representation. The MMAE is a transformer-based architecture, which we train by masking 75% of the data and reconstructing missing image and spectral tokens. We use this model to test three applications: spectral and image reconstruction from heavily masked data and redshift regression from images alone. It recovers key physical features, such as galaxy shapes, atomic emission line peaks, and broad continuum slopes, though it struggles with fine image details and line strengths. For redshift regression, the MMAE performs comparably or better than prior multi-modal models in terms of prediction scatter even when missing spectra in testing. These results highlight both the potential and limitations of masked autoencoders in astrophysics and motivate extensions to additional modalities, such as text, for foundation models.
Paper Structure (9 sections, 3 figures, 1 table)

This paper contains 9 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Data
  5. Methods
  6. Results and discussion
  7. Feature reconstruction
  8. Redshift regression
  9. Conclusion

Figures (3)

  • Figure 1: The model's reconstruction process is shown for a low redshift source with 75% masking of both modalities. We measure the peak location, amplitude, and width of H-$\alpha$ in the augmented and generated spectra. The H-$\alpha$ line has an observed center at 7042.8 Å with a height of 3.04 and a width of 34.5 Å, while the model reconstructed it at 7066.8 Å with a height of 0.62 and a width of 528 Å.
  • Figure 2: The model's reconstruction process is shown for a high redshift source with a fully masked spectrum and a fully unmasked image. We measure the peak location, amplitude, and width of Lyman-$\alpha$ and C IV in the augmented and generated spectra. The Lyman-$\alpha$ line has an observed center at 3851.6 Å with a height of 17.24 and a width of 48 Å, compared to a reconstructed center at 3923.6 Å, height 5.84, and width 312 Å. Similarly, the C IV line has an observed center at 4907.6 Å, height 7.07, and width 72 Å, while the reconstructed line is at 4931.6 Å, with height 2.48 and width 648 Å.
  • Figure 3: The model's redshift regression results for the entire redshift range are shown (right). The redshift predictions were obtained from test data that had 25% of the image masked and 100% of the spectrum masked. The low-redshift regime used for comparison to AstroCLIP parker_astroclip_2024 is shown in more detail in the bottom left. The top left panel shows the scatter of the MMAE compared to AstroCLIP and a BCNN model jones_redshift_2024 for this low-redshift regime. Lower scatter corresponds to more precise predictions.