Galaxy Zoo Evo: 1 million human-annotated images of galaxies

TL;DR

Galaxy Zoo Evo addresses the need for realistic benchmarks that capture label uncertainty and cross-telescope domain shifts in astronomy. It constructs a Core dataset of 823k galaxy images with 104M crowdsourced labels across four telescopes, and four Downstream datasets (including Euclid, strong lenses, rings, and faint debris) for focused evaluation. Using multinomial losses on per-question vote counts and high-confidence class aggregations, the authors benchmark several modern architectures and demonstrate surprising trends in generalization and fine-tuning across domains. The dataset is designed as a living resource to test domain adaptation, continual learning, and efficient crowdsourcing, with broad impact for building robust foundation models in astronomy.

Abstract

We introduce Galaxy Zoo Evo, a labeled dataset for building and evaluating foundation models on images of galaxies. GZ Evo includes 104M crowdsourced labels for 823k images from four telescopes. Each image is labeled with a series of fine-grained questions and answers (e.g. "featured galaxy, two spiral arms, tightly wound, merging with another galaxy"). These detailed labels are useful for pretraining or finetuning. We also include four smaller sets of labels (167k galaxies in total) for downstream tasks of specific interest to astronomers, including finding strong lenses and describing galaxies from the new space telescope Euclid. We hope GZ Evo will serve as a real-world benchmark for computer vision topics such as domain adaption (from terrestrial to astronomical, or between telescopes) or learning under uncertainty from crowdsourced labels. We also hope it will support a new generation of foundation models for astronomy; such models will be critical to future astronomers seeking to better understand our universe.

Figures (8)

• Figure 1: Overview of GZ Evo. GZ Evo includes a large-scale 'Core' dataset with fine-grained annotations describing the typical features of 823k galaxies, and four smaller 'Downstream' datasets with annotations of specific scientific interest.
• Figure 2: Three illustrative workflows using GZ Evo. Supervised transfer: train a supervised model on 'Core' (galaxy images with general descriptions) and finetune on 'Downstream'. Generalization evaluation: test an existing model on 'Core' and 'Downstream'. Continual learning: finetune an existing model on 'Core', and then finetune again to maximise performance on 'Downstream'
• Figure 3: Left: Class counts by subset, when using our aggregated high-confidence class labels. Colours normalized by subset (row) to show the varying class distribution within each subset. This is a real(istic) test scenario for domain adaption. Right: an example galaxy image for each survey and class.
• Figure 4: Test loss vs. dataset size when training on the GZ Evo Core dataset and then finetuning on the GZ Evo Downstream datasets. Mean of six seeds. ConvNeXt-Base performs best at GZ Rings, GZ Euclid, and overall. SoViT-400m/14 performs best at finding strong lenses. Small modern models (MaxViT-Tiny and ConvNeXt-Nano) perform best at our smaller Is Debris dataset.
• Figure 5: Distribution of the number of volunteer annotators per galaxy, per campaign. Most campaigns have approximately 40 annotators per galaxy. GZ DESI has a bimodal distribution with approximately half of galaxies receiving approximately 40 votes, and the remainder receiving approximately 5; see below. Legend shows the total number of completed decision tree annotations (i.e. the sum of the above) per campaign.