Autoregressive deep learning for real-time simulation of soft tissue dynamics during virtual neurosurgery

TL;DR

This work tackles the challenge of real-time, high-fidelity brain deformation during neurosurgical simulation by introducing an autoregressive mesh-based surrogate built on Universal Physics Transformers. The model predicts successive brain displacements under instrument interaction, trained with a stochastic teacher forcing schedule to stabilize long-horizon rollouts. It demonstrates strong accuracy, generalization to unseen instrument dynamics, and real-time performance (≈18 ms per step on consumer hardware), while scaling to meshes with over 150k nodes and enabling integration into interactive simulators. The results indicate that transformer-based surrogates can provide rapid, smooth, and physically plausible brain tissue dynamics, offering a foundation for realistic, hands-on neurosurgical training environments.

Abstract

Accurate simulation of brain deformation is a key component for developing realistic, interactive neurosurgical simulators, as complex nonlinear deformations must be captured to ensure realistic tool-tissue interactions. However, traditional numerical solvers often fall short in meeting real-time performance requirements. To overcome this, we introduce a deep learning-based surrogate model that efficiently simulates transient brain deformation caused by continuous interactions between surgical instruments and the virtual brain geometry. Building on Universal Physics Transformers, our approach operates directly on large-scale mesh data and is trained on an extensive dataset generated from nonlinear finite element simulations, covering a broad spectrum of temporal instrument-tissue interaction scenarios. To reduce the accumulation of errors in autoregressive inference, we propose a stochastic teacher forcing strategy applied during model training. Specifically, training consists of short stochastic rollouts in which the proportion of ground truth inputs is gradually decreased in favor of model-generated predictions. Our results show that the proposed surrogate model achieves accurate and efficient predictions across a range of transient brain deformation scenarios, scaling to meshes with up to 150,000 nodes. The introduced stochastic teacher forcing technique substantially improves long-term rollout stability, reducing the maximum prediction error from 6.7 mm to 3.5 mm. We further integrate the trained surrogate model into an interactive neurosurgical simulation environment, achieving runtimes below 10 ms per simulation step on consumer-grade inference hardware. Our proposed deep learning framework enables rapid, smooth and accurate biomechanical simulations of dynamic brain tissue deformation, laying the foundation for realistic surgical training environments.

Figures (11)

• Figure 1: Overview of our framework for autoregressive prediction of brain tissue deformation. At each time step, the model receives the previously predicted displacement and the current collision field as input. The overlaid arrows on the collision fields indicate the surgical instrument’s pushing direction.
• Figure 2: Prediction of the displacement field at the next time step. The current displacement field is concatenated with the respective collision field. Spatial context is incorporated through positional encodings after passing through a learned embedding layer. The input is then mapped into latent space via a message passing layer, that aggregates information at randomly selected supernodes. This latent signal is propagated forward through a stack of transformer blocks and evaluated at the mesh coordinates using perceiver-style cross-attention.
• Figure 3: Stochastic teacher forcing step for a rollout with window size $S_{\text{STF}}=3$. At each prediction step the input is sampled as either the numerically precomputed ground truth $u^{t_i}$ (teacher-forced) or the predicted value $\hat{u}^{t_i}$ (autoregressive), according to the probability $p$. The first prediction is excluded for the loss computation. Collision fields are omitted for clarity.
• Figure 4: Visualization of the predefined collision domain (green) around the Sylvian fissure, with an example of a sampled collision site and direction (blue) and a superimposed surgical instrument (spatula) demonstrating a typical interaction scenario.
• Figure 5: Example trajectory of a time-dependent push factor, with selected displacement fields overlaid at corresponding time steps to illustrate the resulting deformation progression.