Monte Carlo conformal prediction for quantifying uncertainty in radio galaxy classification under ambiguous ground truth

Abstract

Dramatically increasing data volumes are forcing astronomers to adopt automated methods for the identification and classification of astronomical objects. Although deep-learning models are often well-suited to this task, obtaining a measure of uncertainty on their predictions is challenging. Here we consider the suitability of Monte Carlo conformal prediction (MCCP) set size and confidence as measures of model uncertainty for the astronomical classification of radio galaxies. We demonstrate this approach using model predictions from a pre-trained radio galaxy foundation model, fine-tuned on a smaller set of labelled radio galaxies. We calibrate the MCCP by obtaining annotator-derived soft label distributions, i.e. probability distributions over classes instead of single class assignments, for each of these labelled radio galaxies and compare the resulting set sizes and confidence scores to predictive entropy measures for each galaxy obtained using a supervised Bayesian deep-learning model trained using Hamiltonian Monte Carlo (HMC). The comparison reveals only a weak correlation between the measures.

Figures (9)

• Figure 1: Radio images from the FIRST sky survey VLA-FIRST depicting (a) an FRI galaxy, (b) an FRII galaxy, and (c) a hybrid galaxy. Each image has dimensions of 150$\mathrm{\times}$150 pixels, with each pixel corresponding to an angular size of 1.8 arcseconds.
• Figure 2: An example from the Zooniverse survey, with the FIRST image on the left and the WISE image on the right. Although both are greyscale images, they are both shown with a colour map to make their features more easily distinguishable.
• Figure 3: A heatmap showing the vote fraction between annotators as a percentage (left), and a plot showing the number of annotations provided by each annotator (right).
• Figure 4: Embedding of MiraBest data points from the model finetuned on majority-voted labels, colour-coded by (a) majority-vote annotator label and (b) model prediction. Points that were included in the training set are plotted with lower opacity. Note that the x and y dimensions here have no physical interpretation.
• Figure 5: Embedding of MiraBest data points from the model finetuned on the original MiraBest labels, colour-coded by (a) majority-vote annotator label and (b) model prediction. Note that the x and y dimensions here have no physical meaning.