High-Precision Geosteering via Reinforcement Learning and Particle Filters

TL;DR

This paper tackles the challenge of real-time, robust geosteering by integrating reinforcement learning (RL) with particle filtering (PF) to handle uncertainty in subsurface boundaries. It proposes three decision-making pathways—RL alone, PF alone, and a synergistic RL+PF approach (RL-Estimation)—plus a rule-based PF-informed method for comparison. Through a realistic geosteering scenario with gamma-ray logs and multiple layers and faults, RL-Estimation achieves the best performance, yielding reservoir contact near 89% and high stability, albeit with substantial computational cost due to PF. Benchmarking against theoretical optima shows PF-based RL approaches can closely match the best possible outcomes when state estimates are accurate, while the look-ahead information offers the greatest gains. Overall, the study demonstrates a meaningful synergy between RL and PF for high-precision geosteering and highlights directions for reducing computational burden while preserving accuracy.

Abstract

Geosteering, a key component of drilling operations, traditionally involves manual interpretation of various data sources such as well-log data. This introduces subjective biases and inconsistent procedures. Academic attempts to solve geosteering decision optimization with greedy optimization and Approximate Dynamic Programming (ADP) showed promise but lacked adaptivity to realistic diverse scenarios. Reinforcement learning (RL) offers a solution to these challenges, facilitating optimal decision-making through reward-based iterative learning. State estimation methods, e.g., particle filter (PF), provide a complementary strategy for geosteering decision-making based on online information. We integrate an RL-based geosteering with PF to address realistic geosteering scenarios. Our framework deploys PF to process real-time well-log data to estimate the location of the well relative to the stratigraphic layers, which then informs the RL-based decision-making process. We compare our method's performance with that of using solely either RL or PF. Our findings indicate a synergy between RL and PF in yielding optimized geosteering decisions.

Figures (12)

• Figure 1: RL agent interaction with a decision-making environment muhammad2023optimal. A decision-making agent interacts with a decision-making environment by receiving state $s_t$ and reward $r_t$ inputs, taking action $a_t$, and receiving feedback $s_{t+1}$ and $r_{t+1}$ in response
• Figure 2: Illustration of using well-log data as inputs for PF to estimate the distance to reservoir boundaries. The solid black lines show the true reservoir boundaries, while the dashed black lines show the estimated boundaries derived from PF. The red lines show the trajectory of the drilled well, while the red arrows show the decision-making criterion. The figure on the left illustrates the RL-Log method, while the complete illustration refers to the RL-Estimation method.
• Figure 3: Illustration of the rule-based decision-making method. It shows the use of a PF particle as an initial reference to generate look-ahead estimates. The estimates are shown as dashed blue lines. Red arrows show distances to estimated look-ahead boundaries from the projected well, which are shown as dashed blue lines.
• Figure 4: Gamma-ray log data based on logalyaev2022.
• Figure 5: Illustration of the geosteering decision-making scenario. The trajectory of the drilled well is shown as a dotted red line, with each point indicating a decision point. Reservoir boundaries are randomly generated using functions from Alyaev (2022) and shown by black lines. The separate blue line plot shows the gamma-ray log data.