Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pervasive Label Errors in Seismological Machine Learning Datasets

Albert Leonardo Aguilar Suarez, Gregory Beroza

TL;DR

The paper tackles the problem that seismological ML datasets contain label errors that degrade performance. It applies an ensemble of PhaseNet and EQTransformer to eight prominent datasets, categorizing errors into unlabeled earthquakes, earthquakes in noise samples, false earthquakes, and inaccurate timing, and reports an average label-error rate of $3.9\%$. This work highlights that data quality is a primary driver of model reliability and proposes a data-centric path—flagging, fixing, and augmenting training data, with semi-supervised and manual oversight—to improve downstream tasks like catalogs and phase picking. The findings underscore the need for better data curation and benchmarks (e.g., for catalog building, phase association, and event location) to sustain reliable seismic ML systems in diverse settings.

Abstract

The recent boom in artificial intelligence and machine learning has been powered by large datasets with accurate labels, combined with algorithmic advances and efficient computing. The quality of data can be a major factor in determining model performance. Here, we detail observations of commonly occurring errors in popular seismological machine learning datasets. We used an ensemble of available deep learning models PhaseNet and EQTransformer to evaluate the dataset labels and found four types of errors ranked from most prevalent to least prevalent: (1) unlabeled earthquakes; (2) noise samples that contain earthquakes; (3) inaccurately labeled arrival times, and (4) absent earthquake signals. We checked a total of 8.6 million examples from the following datasets: Iquique, ETHZ, PNW, TXED, STEAD, INSTANCE, AQ2009, and CEED. The average error rate across all datasets is 3.9 %, ranging from nearly zero to 8 % for individual datasets. These faulty data and labels are likely to degrade model training and performance. By flagging these errors, we aim to increase the quality of the data used to train machine learning models, especially for the measurement of arrival times, and thereby to improve the reliability of the models. We present a companion list of examples that contain problems, aiming to integrate them into training routines so that only the reliable data is used for training.

Pervasive Label Errors in Seismological Machine Learning Datasets

TL;DR

The paper tackles the problem that seismological ML datasets contain label errors that degrade performance. It applies an ensemble of PhaseNet and EQTransformer to eight prominent datasets, categorizing errors into unlabeled earthquakes, earthquakes in noise samples, false earthquakes, and inaccurate timing, and reports an average label-error rate of . This work highlights that data quality is a primary driver of model reliability and proposes a data-centric path—flagging, fixing, and augmenting training data, with semi-supervised and manual oversight—to improve downstream tasks like catalogs and phase picking. The findings underscore the need for better data curation and benchmarks (e.g., for catalog building, phase association, and event location) to sustain reliable seismic ML systems in diverse settings.

Abstract

The recent boom in artificial intelligence and machine learning has been powered by large datasets with accurate labels, combined with algorithmic advances and efficient computing. The quality of data can be a major factor in determining model performance. Here, we detail observations of commonly occurring errors in popular seismological machine learning datasets. We used an ensemble of available deep learning models PhaseNet and EQTransformer to evaluate the dataset labels and found four types of errors ranked from most prevalent to least prevalent: (1) unlabeled earthquakes; (2) noise samples that contain earthquakes; (3) inaccurately labeled arrival times, and (4) absent earthquake signals. We checked a total of 8.6 million examples from the following datasets: Iquique, ETHZ, PNW, TXED, STEAD, INSTANCE, AQ2009, and CEED. The average error rate across all datasets is 3.9 %, ranging from nearly zero to 8 % for individual datasets. These faulty data and labels are likely to degrade model training and performance. By flagging these errors, we aim to increase the quality of the data used to train machine learning models, especially for the measurement of arrival times, and thereby to improve the reliability of the models. We present a companion list of examples that contain problems, aiming to integrate them into training routines so that only the reliable data is used for training.

Paper Structure

This paper contains 8 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Errors
  3. Unlabeled Earthquakes
  4. Earthquakes in Noise Samples
  5. False Earthquakes
  6. Inaccurate Timing
  7. From Big Data to Good Data
  8. Future Directions

Figures (4)

  • Figure 1: Samples that contain more earthquakes than those labeled. Dotted lines indicate modelensemble inferred arrivals.
  • Figure 2: Noise examples for which earthquake arrivals were found. PhaseNet and EQTransformer arrivals are shown with the dotted lines.
  • Figure 3: Labeled arrivals for which there are no earthquake waveforms.
  • Figure 4: Samples with inaccurately timed arrivals. Dotted lines show the machine learning ensemble. The name of the dataset is indicated on the title of each panel.