Synergizing Multigrid Algorithms with Vision Transformer: A Novel Approach to Enhance the Seismic Foundation Model

TL;DR

The paper tackles the challenge of pretraining seismic foundation models with Vision Transformers by explicitly capturing both high- and low-frequency seismogram information. It introduces ADATG, an adaptive two-grid training framework that combines spectrum decomposition with hierarchical Hilbert encoding to feed frequency-targeted patches into a ViT, guided by the frequency principle. Key contributions include a discrete Fourier-based frequency split ($k_0$), a two-grid Hilbert encoding scheme, and an adaptive loss that shifts emphasis from low to high frequencies during training, yielding faster convergence and substantial performance gains on reconstruction tasks. The approach advances seismic visual foundation modeling and can be extended to 3D seismograms and broader tasks in geophysics and waveform inversion.

Abstract

Due to the emergency and homogenization of Artificial Intelligence (AI) technology development, transformer-based foundation models have revolutionized scientific applications, such as drug discovery, materials research, and astronomy. However, seismic data presents unique characteristics that require specialized processing techniques for pretraining foundation models in seismic contexts with high- and low-frequency features playing crucial roles. Existing vision transformers (ViTs) with sequential tokenization ignore the intrinsic pattern and fail to grasp both the high- and low-frequency seismic information efficiently and effectively. This work introduces a novel adaptive two-grid foundation model training strategy (ADATG) with Hilbert encoding specifically tailored for seismogram data, leveraging the hierarchical structures inherent in seismic data. Specifically, our approach employs spectrum decomposition to separate high- and low-frequency components and utilizes hierarchical Hilbert encoding to represent the data effectively. Moreover, observing the frequency principle observed in ViTs, we propose an adaptive training strategy that initially emphasizes coarse-level information and then progressively refines the model's focus on fine-level features. Our extensive experiments demonstrate the effectiveness and efficiency of our training methods. This research highlights the importance of data encoding and training strategies informed by the distinct characteristics of high- and low-frequency features in seismic images, ultimately contributing to the enhancement of visual seismic foundation models pretraining.

Figures (12)

• Figure 1: A toy example demonstrating frequency principle.
• Figure 2: Pipeline of Seismic Foundation Model Pretraining with ADATG. (A) Input seismogram spectral decomposition via Fourier transform, (B) Frequency-adaptive Hilbert encoding (fine/coarse grids for high/low frequencies), (C) Vision Transformer training with adaptive MAE strategy, (D) Frequency-adaptive Hilbert decoding with inverse transformer matrix, (E) Merged reconstruction of difference frequency components.
• Figure 3: Reconstructed images using different pretrained Seismic Foundation Models (SFM). From left to right: the original image, reconstruction using the base Vision Transformer (ViT) architecture sheng2025seismic, the Hilbert Encoding ViT (HE-ViT), the randomized Two-grid method (Ran-TG), and two variants of ADATG: one incorporating both high and low frequency components with Hilbert Encoding (ADATG-HH), and another using only high-frequency components (ADATG-NH). All methods are applied under identical evaluation conditions to enable direct visual comparison.
• Figure 4: Hilbert Encoding (Left Two) and Twogrid Hilbert Encoding (Right Two).
• Figure 5: Training Dynamics.

Theorems & Definitions (3)

• Definition 1: Discrete Fourier Transform in Matrix Form trefethen1997numerical
• Definition 2
• Definition 3