Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeisLM: a Foundation Model for Seismic Waveforms

Tianlin Liu, Jannes Münchmeyer, Laura Laurenti, Chris Marone, Maarten V. de Hoop, Ivan Dokmanić

Abstract

We introduce the Seismic Language Model (SeisLM), a foundational model designed to analyze seismic waveforms -- signals generated by Earth's vibrations such as the ones originating from earthquakes. SeisLM is pretrained on a large collection of open-source seismic datasets using a self-supervised contrastive loss, akin to BERT in language modeling. This approach allows the model to learn general seismic waveform patterns from unlabeled data without being tied to specific downstream tasks. When fine-tuned, SeisLM excels in seismological tasks like event detection, phase-picking, onset time regression, and foreshock-aftershock classification. The code has been made publicly available on https://github.com/liutianlin0121/seisLM.

SeisLM: a Foundation Model for Seismic Waveforms

Abstract

We introduce the Seismic Language Model (SeisLM), a foundational model designed to analyze seismic waveforms -- signals generated by Earth's vibrations such as the ones originating from earthquakes. SeisLM is pretrained on a large collection of open-source seismic datasets using a self-supervised contrastive loss, akin to BERT in language modeling. This approach allows the model to learn general seismic waveform patterns from unlabeled data without being tied to specific downstream tasks. When fine-tuned, SeisLM excels in seismological tasks like event detection, phase-picking, onset time regression, and foreshock-aftershock classification. The code has been made publicly available on https://github.com/liutianlin0121/seisLM.

Paper Structure

This paper contains 42 sections, 3 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Supervised-learning models for seismic tasks.
  4. Unsupervised learning models for seismic tasks.
  5. Foundation models for seismic tasks and their relationships to our work.
  6. Seismic Language Model
  7. SeisLM architecture
  8. Model input.
  9. ConvNet encoder.
  10. Transformer encoder.
  11. Quantization.
  12. Training
  13. Pretraining setup
  14. Masking.
  15. Contrastive loss.
  16. ...and 27 more sections

Figures (8)

  • Figure 1: Illustration of the self-supervised learning of Seismic Language Model (SeisLM). A ConvNet encodes raw 3-channel seismic waveforms from a single station into a feature sequence. The model then follows two paths. In the lower path, we apply random masking to these waveform features before passing them to a transformer. The transformer aims to reconstruct aspects of the masked convolutional features. In the upper path, we prepare the reconstruction targets: continuous-valued convolutional features are discretized into a sequence of vectors with a finite vocabulary size through vector quantization (VQ; Van2017neuralRazavi2019generating). This overall model closely resembles Wav2vec2 Baevski2020wav2vec for audio self-supervised learning.
  • Figure 2: Pretraining loss of SeisLM.
  • Figure 3: t-SNE embeddings of SeisLM features. Compared to a randomly initialized SeisLM-base (left panel), a self-supervised SeisLM-base (right panel) separate the embeddings of earthquake and noise waveforms more effectively.
  • Figure 4: Performance of models on the event detection task. Each panel indicates a finetuning dataset. The $x$ axis indicate the fraction of training dataset; the $y$ axis shows the AUC metric: it represents the area under the curve that plots the true positive rate against the false positive rate at various threshold levels for a binary classification task.
  • Figure 5: Performance of models on the phase identification task.
  • ...and 3 more figures