Diffusion Autoencoders with Perceivers for Long, Irregular and Multimodal Astronomical Sequences

TL;DR

This work tackles self-supervised representation learning for long, irregular, and multimodal sequences. It introduces the Diffusion AutoEncoder with Perceivers (daep), which combines a flexible Perceiver encoder with a diffusion-based Perceiver-IO decoder, and benchmarks it against a masked autoencoder baseline (maep) within the same architecture family. Across diverse astronomical datasets, daep achieves lower reconstruction errors, richer latent discriminative structure, and better preservation of high-frequency details than VAE and maep baselines, including successful cross-modality inference with modality dropping. While diffusion decoding is computationally heavier, the approach is domain-agnostic and shows promise for other irregular, heterogeneous data domains such as healthcare and finance. The work provides a practical, strong baseline for fair comparisons in irregular-sequence SSL and highlights the importance of decoder context for capturing fine-grained structure.

Abstract

Self-supervised learning has become a central strategy for representation learning, but the majority of architectures used for encoding data have only been validated on regularly-sampled inputs such as images, audios. and videos. In many scientific domains, data instead arrive as long, irregular, and multimodal sequences. To extract semantic information from these data, we introduce the Diffusion Autoencoder with Perceivers (daep). daep tokenizes heterogeneous measurements, compresses them with a Perceiver encoder, and reconstructs them with a Perceiver-IO diffusion decoder, enabling scalable learning in diverse data settings. To benchmark the daep architecture, we adapt the masked autoencoder to a Perceiver encoder/decoder design, and establish a strong baseline (maep) in the same architectural family as daep. Across diverse spectroscopic and photometric astronomical datasets, daep achieves lower reconstruction errors, produces more discriminative latent spaces, and better preserves fine-scale structure than both VAE and maep baselines. These results establish daep as an effective framework for scientific domains where data arrives as irregular, heterogeneous sequences.

Figures (16)

• Figure 1: Schematic of tokenizers for general irregularly measured sequences.
• Figure 2: Schematic of the perceiver encoder and perceiver-IO score/decoder model used in daep and mae.
• Figure 3: Two example reconstructions of variable star spectra. daep captures finer spectroscopic features, while the VAE mainly reproduces the continuum, likely due to posterior collapse. Both daep and maep successfully capture small-scale spectral features.
• Figure 4: Residuals for test spectra in the LAMOST dataset. Both daep and maep show smaller residuals than VAE. daep also preserves higher-resolution structure, with fewer residual spikes (e.g., near 7000 Å) compared to maep. The red line marks 0 residual while the orange line marks a residual of $\pm$0.1.
• Figure 5: Reconstruction of SEDM spectra for ZTF supernovae from four latent tokens of dimension four using daep and baseline models.