Supervised Pretraining for Material Property Prediction

Abstract

Accurate prediction of material properties facilitates the discovery of novel materials with tailored functionalities. Deep learning models have recently shown superior accuracy and flexibility in capturing structure-property relationships. However, these models often rely on supervised learning, which requires large, well-annotated datasets an expensive and time-consuming process. Self-supervised learning (SSL) offers a promising alternative by pretraining on large, unlabeled datasets to develop foundation models that can be fine-tuned for material property prediction. In this work, we propose supervised pretraining, where available class information serves as surrogate labels to guide learning, even when downstream tasks involve unrelated material properties. We evaluate this strategy on two state-of-the-art SSL models and introduce a novel framework for supervised pretraining. To further enhance representation learning, we propose a graph-based augmentation technique that injects noise to improve robustness without structurally deforming material graphs. The resulting foundation models are fine-tuned for six challenging material property predictions, achieving significant performance gains over baselines, ranging from 2% to 6.67% improvement in mean absolute error (MAE) and establishing a new benchmark in material property prediction. This study represents the first exploration of supervised pertaining with surrogate labels in material property prediction, advancing methodology and application in the field.

Figures (7)

• Figure 1: Schematic diagram of the proposed SPMat (Supervised Pretraining for Material Property Prediction) framework. Initially, the material structure undergoes augmentation, including a novel Graph-level Neighbor Distance Noising (GNDN) augmentation, to create diverse views. Deep learning-based encoders and projectors are then used to capture the representations as embeddings, which are utilized in various pretext tasks using labels (top panel). In the downstream task the fine-tuned model starts training on top of the pre-trained encoder and predicts material properties which is a regression task (bottom panel).
• Figure 2: Class-wise distribution of 33,990 stable materials (DFT calculated thermal stability) used in our study , which shows the proportional breakdown across the main attributes of the materials: conductivity, direct bandgap, metallic, and magnetic. The materials were taken from the Materials Project database jain2013commentary. The following attributes of conductivity are characterized as either true or untrue. Conductivity is divided into three categories: conductor, semiconductor, and insulator.
• Figure 3: Distribution of element frequencies in the dataset used in our tests. The most frequent element is oxygen (O), whereas the least common element is neon (Ne). The color scale shows the element frequency, while considerable diversity is indicated by a Shannon entropy of 3.81.
• Figure 4: t-SNE visualization of embeddings from SPMat compared to standard SSL models (Barlow Twins and SimCLR) for bandgap prediction. The left panel shows embeddings from the unsupervised SSL models: Barlow Twins (top) and SimCLR (bottom). The right panel shows SPMat's results for the two variants defined in Eqn.\ref{['simple-BT']} (top) and Eqn.\ref{['simple-sc']} (bottom). Colors represent the three classes obtained by discretizing the bandgap values.
• Figure 5: t-SNE visualization for the supervised Barlow Twins loss, keeping only the on- and off-diagonal portions of the loss for pre-trained and fine-tuned models. The left panel represents the model trained with only the off-diagonal loss, while the right panel shows the model trained with only the on-diagonal loss. In each case, the t-SNE plots for pre-trained models are shown on the top and for fine-tuned models on the bottom.