Propagating the prior from shallow to deep with a pre-trained velocity-model Generative Transformer network

TL;DR

This work tackles the challenge of generating subsurface velocity models by proposing VelocityGPT, a top-down, autoregressive framework that advances beyond uniform-image generators by conditioning generation on shallow priors. The core idea combines a Vector-Quantized Variational Autoencoder (VQ-VAE) to compress velocity models into discrete latent codes with a Generative Pre-trained Transformer (GPT) that learns the distribution of these codes and generates deeper layers autoregressively. The approach supports conditioning on geological class, well data, and structural post-stack images, and demonstrates unconditional and conditioned sampling on the OpenFWI dataset, along with a scalable inference strategy to larger models via patch-based processing. The results show meaningful alignment with priors and highlight both the potential benefits for seismic inversion and the practical challenges (e.g., reconstruction fidelity and edge preservation) that motivate future improvements and integration with inversion workflows.

Abstract

Building subsurface velocity models is essential to our goals in utilizing seismic data for Earth discovery and exploration, as well as monitoring. With the dawn of machine learning, these velocity models (or, more precisely, their distribution) can be stored accurately and efficiently in a generative model. These stored velocity model distributions can be utilized to regularize or quantify uncertainties in inverse problems, like full waveform inversion. However, most generators, like normalizing flows or diffusion models, treat the image (velocity model) uniformly, disregarding spatial dependencies and resolution changes with respect to the observation locations. To address this weakness, we introduce VelocityGPT, a novel implementation that utilizes Transformer decoders trained autoregressively to generate a velocity model from shallow subsurface to deep. Owing to the fact that seismic data are often recorded on the Earth's surface, a top-down generator can utilize the inverted information in the shallow as guidance (prior) to generating the deep. To facilitate the implementation, we use an additional network to compress the velocity model. We also inject prior information, like well or structure (represented by a migration image) to generate the velocity model. Using synthetic data, we demonstrate the effectiveness of VelocityGPT as a promising approach in generative model applications for seismic velocity model building.

Figures (11)

• Figure 1: a). The framework and the network architecture of VelocityGPT. The framework consists of two stages: VQ-VAE training to convert velocity models into their discrete representation and autoregressive GPT training to model the distribution of the velocities. b). A diagram of a single Transformer decoder block.
• Figure 2: Velocity encoding/decoding process. Solid colors in the discrete latent domain (middle) represent the region with the corresponding shaded colors in the velocity domain (left). Dashed boxes represent the region to be generated.
• Figure 3: Two examples of VQ-VAE post-training velocity reconstruction, which corresponds to FlatVelA (a) and CurveFaultA (b) classes.
• Figure 4: Two examples of VQ-VAE post-training post-stack image reconstruction, which corresponds to FlatVelA (a) and CurveFaultA (b) classes.
• Figure 5: Samples from every class, given the input from the shallow part of a randomly picked model from the validation set displayed on the right column.