Is AI Robust Enough for Scientific Research?

TL;DR

The paper demonstrates that high-precision neural networks used in scientific contexts are surprisingly vulnerable to minute perturbations, including adversarial FGSM inputs, across diverse domains such as weather, chemistry, fluid dynamics, QCD, and wireless communications. It combines domain-specific models (FourCastNet, DeePMD-kit, NNfoil-C, DLQP, BMQN) with controlled perturbations to reveal pervasive instability in outputs, often exceeding random-noise effects in targeted directions. The authors discuss whether this fragility is an inherent property of neural networks and propose directions to mitigate it, notably randomized architectures that can smooth the loss landscape and improve robustness. Overall, the work highlights the urgent need for robust and trustworthy AI systems in critical scientific applications, where small perturbations can propagate into substantial decision-making errors.

Abstract

We uncover a phenomenon largely overlooked by the scientific community utilizing AI: neural networks exhibit high susceptibility to minute perturbations, resulting in significant deviations in their outputs. Through an analysis of five diverse application areas -- weather forecasting, chemical energy and force calculations, fluid dynamics, quantum chromodynamics, and wireless communication -- we demonstrate that this vulnerability is a broad and general characteristic of AI systems. This revelation exposes a hidden risk in relying on neural networks for essential scientific computations, calling further studies on their reliability and security.

Figures (6)

• Figure 1: Accurate neural network fails to predict the correct label with small non-random perturbations. The trained neural network is evaluated under three different conditions: normal scenario, with random noise, and under an attack using the Fast Gradient Sign Method (FGSM) goodfellow2014explainingkurakin2016adversarial. In the normal scenario, the network is tested using the original test dataset. For the random noise scenario, the test dataset images are modified by adding random noise with pixel values of either -$\epsilon$ or $\epsilon$, where $\epsilon=0.1$. In the FGSM attack scenario we perturb the input images by adding a small, calculated noise in the direction of the gradient of the loss function. Specifically, the transformation applied is: $x \rightarrow x + \epsilon \cdot \text{sign}(\nabla_{x} l(f(x, \theta), y))$, where $x$ represents the input images. It is important to note that $y$ is the true label which corresponds to an untargeted attack, where the neural network's prediction would deviate away from the true label. $l(f(x, \theta), y)$ is the loss function, and $\theta$ denotes the weights of the trained network. The trained Googlenet model szegedy2015going, when evaluated on the CIFAR-10 dataset Alex2009, achieves an accuracy of 89% on both the original and random noise scenarios but only 18% accuracy on the images subjected to the FGSM attack.
• Figure 2: Testing results of the FourCastNet under perturbations. Upper panel A shows the wind speed distribution at a selected time step. The input (first row) is perturbed using the FGSM attack with an epsilon value of 0.05 (roughly changing the input by 5%), as well as random noise values of -0.05 and 0.05. The corresponding predictions are displayed in the second row, with the ground-truth for this time step shown in the first column of the third row. A specific region in the ground-truth and the FGSM attack prediction is magnified for detailed comparison. Panel B compares the predictive root mean square error (RMSE) between the normal scenario, input perturbed by the FGSM attack, and random noise. The 20 atmospheric variables include surface-level variables such as U10, V10, T2M, SP, and MSLP; variables at 1000 hPa including U, V, and Z; variables at 850 hPa and 500 hPa including T, U, V, Z, and RH; a variable at 50 hPa, which is Z; and an integrated variable, TCWV.
• Figure 3: Training and test results using the DeePMD-kit in calculating the energy and force of the methane molecule. A, the training loss curve of the network. "rmse_val", "rmse_trn", "rmse_e_val", "rmse_e_trn", "rmse_f_val", "rmse_f_trn", refer to the validation loss, training loss, RMS validation error of energy, RMS training error of energy, RMS validation error of force, RMS training error of force. B and C, comparison of MSE between DeePMD-kit predictions and VASP calculations for methane molecules in the test phase. The test is performed on 40 different frames with varying coordinates and box dimensions. For each frame, the coordinates are slightly perturbed by noise values of either -0.05 or 0.05, applied using the FGSM attack (orange) or random noise (green).
• Figure 4: NNfoil-C predictions under different perturbations. Panel A represents the perturbations added to spatial coordinates of airfoil surface sampling points (illustrative only; actual number of sampling points is ten times that shown). B and C compare the airfoil surface pressure coefficient distributions output by NNfoil-C after adding random noise and FGSM attacks to the six cases used in the literature cao2024solving with the reference solutions obtained through the finite volume method (FVM). D gives relative L2 error between the airfoil surface pressure coefficient distributions of the six cases output by the original NNfoil-C and NNfoil-C after two types of attacks and the reference solution.
• Figure 5: Deep-learning quasi-particle model developed to reproduce the QCD equation of state. A, the framework of the neural network, where the input is the one-dimensional temperature and the output is the corresponding quasi-particle mass, contributing to the partition function. B, the pressure and energy density as functions of temperature using the network model as well as the lattice QCD calculations. C and D, the test errors of the predicted pressure and lattice QCD calculations as a function of temperature under 3 different conditions: (1) no perturbations are involved (training accuracy), (2) the temperatures are slightly perturbed with values of either $\epsilon$ or -$\epsilon$, using the FGSM attack (red), and (3) the random noise (orange), for $\epsilon=0.005$ and $\epsilon=0.01$.