Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniSpectra: A Unified Foundation Model for Native Resolution Astronomical Spectra

Md Khairul Islam, Judy Fox

TL;DR

OmniSpectra introduces a native-resolution, transformer-based foundation model for astronomical spectra that accepts variable-length inputs without resampling. It uses adaptive patching, validity-aware attention, and a hybrid sinusoidal wavelength encoding to learn from eight diverse surveys in a self-supervised manner. Through masked reconstruction and downstream linear probes, it achieves state-of-the-art performance on galaxy and stellar property estimation, source classification, and redshift prediction, with strong zero-shot and few-shot generalization. This approach enables unified, scalable spectral representation learning across instruments, reducing the need for task-specific models while preserving physically meaningful spectral structure. The work advances universal representation learning in astronomy by leveraging large unlabeled spectral data across heterogeneous instruments.

Abstract

We present OmniSpectra, the first native-resolution foundation model for astronomy spectra. Unlike traditional models, which are limited to fixed-length input sizes or configurations, OmniSpectra handles spectra of any length at their original size, without resampling or interpolation. Despite the large-scale spectroscopic data from diverse surveys fueling the rapid growth of astronomy, existing foundation models are limited to a fixed wavelength range and specific instruments. OmniSpectra is the first foundation model to learn simultaneously from multiple real-world spectra surveys with different configurations at a large scale. We achieve this by designing a novel architecture with adaptive patching across variable lengths, sinusoidal global wavelength encoding, local positional embeddings through depthwise convolution, and validity-aware self-attention masks. Allowing us to learn multi-scale spatial patterns while skipping attention for invalid patches. Even with a limited training example, OmniSpectra demonstrates excellent zero-shot generalization compared to methods tailored for specific tasks. This transfer learning capability makes this model the state-of-the-art across various astronomy tasks, including source classification, redshift estimation, and properties prediction for stars and galaxies. OmniSpectra reduces the need for training individual models for different tasks from scratch, establishing itself as the next-generation astronomy foundation model.

OmniSpectra: A Unified Foundation Model for Native Resolution Astronomical Spectra

TL;DR

OmniSpectra introduces a native-resolution, transformer-based foundation model for astronomical spectra that accepts variable-length inputs without resampling. It uses adaptive patching, validity-aware attention, and a hybrid sinusoidal wavelength encoding to learn from eight diverse surveys in a self-supervised manner. Through masked reconstruction and downstream linear probes, it achieves state-of-the-art performance on galaxy and stellar property estimation, source classification, and redshift prediction, with strong zero-shot and few-shot generalization. This approach enables unified, scalable spectral representation learning across instruments, reducing the need for task-specific models while preserving physically meaningful spectral structure. The work advances universal representation learning in astronomy by leveraging large unlabeled spectral data across heterogeneous instruments.

Abstract

We present OmniSpectra, the first native-resolution foundation model for astronomy spectra. Unlike traditional models, which are limited to fixed-length input sizes or configurations, OmniSpectra handles spectra of any length at their original size, without resampling or interpolation. Despite the large-scale spectroscopic data from diverse surveys fueling the rapid growth of astronomy, existing foundation models are limited to a fixed wavelength range and specific instruments. OmniSpectra is the first foundation model to learn simultaneously from multiple real-world spectra surveys with different configurations at a large scale. We achieve this by designing a novel architecture with adaptive patching across variable lengths, sinusoidal global wavelength encoding, local positional embeddings through depthwise convolution, and validity-aware self-attention masks. Allowing us to learn multi-scale spatial patterns while skipping attention for invalid patches. Even with a limited training example, OmniSpectra demonstrates excellent zero-shot generalization compared to methods tailored for specific tasks. This transfer learning capability makes this model the state-of-the-art across various astronomy tasks, including source classification, redshift estimation, and properties prediction for stars and galaxies. OmniSpectra reduces the need for training individual models for different tasks from scratch, establishing itself as the next-generation astronomy foundation model.
Paper Structure (25 sections, 21 equations, 2 figures, 7 tables)

This paper contains 25 sections, 21 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Foundation Models in Astronomy
  3. OmniSpectra
  4. Tokenization
  5. Masking
  6. Embedding
  7. Data Embedding:
  8. Sinusoidal Wavelength Encoding:
  9. Transformer Block
  10. Local Positional Embedding
  11. Reconstruction Loss
  12. Downstream Linear Probe
  13. Experimental Setup
  14. Dataset
  15. Self-Supervised Pretraining
  16. ...and 10 more sections

Figures (2)

  • Figure 1: OmniSpectra: model architecture overview. Left: Spectra tokenization and masking steps. Middle: The Masked Transformer architecture with wavelength and local positional embedding. Right: Downstream tasks
  • Figure 2: Few-shot redshift prediction for galaxy spectrum (here $Z$ represents redshift). OmniSpectra outperforms the current best models in R2-score ($\uparrow$).