AION-1: Omnimodal Foundation Model for Astronomical Sciences

TL;DR

AION-1 introduces a billion-parameter family of omnimodal foundation models for astronomy that unifies 39 data modalities from five major public surveys through bespoke tokenizers and a transformer-based multimodal masked modeling objective. By pretraining on Legacy, HSC, SDSS, DESI, and Gaia with a single frozen encoder, the model achieves strong performance across galaxy and stellar property estimation, morphology classification, segmentation, and spectral super-resolution, while excelling in low-data regimes and enabling zero-shot rare object retrieval. The work demonstrates emergent capabilities such as cross-modal generation and survey-to-survey transfer in embedding space, and provides a scalable blueprint for multimodal scientific foundation models that can integrate instrument-specific noise and heterogeneity. All code, tokenizers, pretrained weights, and evaluation tools are open-sourced to accelerate adoption across astronomy and related data-rich sciences.

Abstract

While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In this paper, we present AION-1, a family of large-scale multimodal foundation models for astronomy. AION-1 integrates heterogeneous imaging, spectroscopic, and scalar data using a two-stage architecture: modality-specific tokenization followed by transformer-based masked modeling of cross-modal token sequences. The model is pretrained on five large-scale surveys: Legacy Survey, Hyper Suprime-Cam (HSC), Sloan Digital Sky Survey (SDSS), Dark Energy Spectroscopic Instrument (DESI), and Gaia. These span more than 200 million observations of stars, galaxies, and quasars. With a single frozen encoder, AION-1 achieves strong results on a broad suite of downstream tasks, including galaxy and stellar property estimation, galaxy morphology classification, similarity-based retrieval, galaxy image segmentation, and spectral super-resolution. We release AION-1 model variants ranging from 300 M to 3.1 B parameters. Beyond astronomy, AION-1 provides a scalable blueprint for multimodal scientific foundation models that can seamlessly integrate noisy, instrument-specific observations. All code, tokenizers, pretrained weights, and a lightweight evaluation suite are released under an open-source license.

Figures (12)

• Figure 1: AION-1 integrates 39 different data modalities — multiband images, optical spectra, and various properties and measurements — into a single model usable for a wide range of downstream applications. It implements a two-step process: first, bespoke tokenization strategies that homogenize the diverse scientific data, followed by multimodal masked modeling that learns how different observations relate, inducing a deep understanding of the underlying physical objects. Astronomers can then leverage AION-1’s rich astrophysical understanding for a variety of downstream tasks.
• Figure 2: AION-1 pre-training inputs. Left: visual representation of the various surveys used during pretraining, their total volume (in number of objects), and the size of the cross-matches between datasets. Right: A more detailed break-down of the modalities provided by each survey.
• Figure 3: Tokenizer Codebook Scaling: We present the image tokenizer reconstruction error (MSE) as a function of FSQ quantizer size. We choose $2^{12}$ as our ultimate codebook size for images, as it reconstruction loss appears to plateau around this point.
• Figure 4: Image Tokenizer Performance: Reconstruction quality of the image tokenizer on three representative Legacy Survey images. The columns show, left to right, the original image, the reconstruction from the tokens, the reconstruction from the Legacy Survey tractor, the $r$-band residual for the tokenizer, and the $r$-band residual for the tractor.
• Figure 5: Spectrum Tokenizer Performance: Reconstruction quality of the spectrum tokenizer on two representative DESI EDR spectra. Top panels show the normalized flux as a function of wavelength, with original input (black) and reconstructed output (red) overlaid. Bottom panels show the residuals in units of the input uncertainty ($\sigma$).