A comparative study of transformer models and recurrent neural networks for path-dependent composite materials

TL;DR

A systematic comparison between RNNs and transformer models trained on sequences of homogenized response of SFRC RVEs shows that while transformer models remain competitive in terms of accuracy on large datasets, the RNNs demonstrate better accuracy on small datasets and show better extrapolation performance.

Abstract

Accurate modeling of Short Fiber Reinforced Composites (SFRCs) remains computationally expensive for full-field simulations. Data-driven surrogate models using Artificial Neural Networks (ANNs) have been proposed as an efficient alternative to numerical modeling, where Recurrent Neural Networks (RNNs) are increasingly being used for path-dependent multiscale modeling by predicting the homogenized response of a Representative Volume Element (RVE). However, recently, transformer models have been developed and they offer scalability and efficient parallelization, yet have not been systematically compared with RNNs in this field. In this study, we perform a systematic comparison between RNNs and transformer models trained on sequences of homogenized response of SFRC RVEs. We study the effect on two types of hyperparameters, namely architectural hyperparameters (such as the number of GRU layers, hidden size, number of attention heads, and encoder blocks) and training hyperparameters (such as learning rate and batch size). Both sets of hyperparameters are tuned using Bayesian optimization. We then analyze scaling laws with respect to dataset size and inference accuracy in interpolation and extrapolation regimes. The results show that while transformer models remain competitive in terms of accuracy on large datasets, the RNNs demonstrate better accuracy on small datasets and show better extrapolation performance. Furthermore, under extrapolation, there is a clear difference, where the RNN remains accurate, while the transformer model performs poorly. On the other hand, the transformer model is 7 times faster at inference, requiring 0.5 ms per prediction compared to the 3.5 ms per prediction for the RNN model.

Figures (14)

• Figure 1: Fiber direction vector $\boldsymbol{p}$ expressed in spherical coordinates $(\theta,\phi)$.
• Figure 2: Example RVEs with their corresponding orientation tensor $\boldsymbol{a}$.
• Figure 3: Schematic of a GRU-based recurrent neural network.
• Figure 4: Schematic of the internal computation within a GRU cell, showing update gate, reset gate, and candidate state.
• Figure 5: Schematic of the Transformer encoder architecture.