MatMMFuse: Multi-Modal Fusion model for Material Property Prediction

TL;DR

MatMMFuse addresses the limited generalization of single-modality crystal-property predictors by fusing structure-aware graph embeddings with global textual context from a pretrained language model via multi-head cross-attention. Trained end-to-end on the Materials Project dataset, it improves prediction across formation energy, band gap, energy above hull, and Fermi energy, and demonstrates strong zero-shot performance on Perovskites, Chalcogenides, and JARVIS subsets. The work shows that combining local structural information with global domain knowledge yields a richer representation, with ablation studies supporting the benefits of encoded knowledge, lattice encoding, and cross-modal attention. Its practical impact lies in enabling more accurate predictions with limited data, supporting industrial applications where collecting extensive training data is expensive.

Abstract

The recent progress of using graph based encoding of crystal structures for high throughput material property prediction has been quite successful. However, using a single modality model prevents us from exploiting the advantages of an enhanced features space by combining different representations. Specifically, pre-trained Large language models(LLMs) can encode a large amount of knowledge which is beneficial for training of models. Moreover, the graph encoder is able to learn the local features while the text encoder is able to learn global information such as space group and crystal symmetry. In this work, we propose Material Multi-Modal Fusion(MatMMFuse), a fusion based model which uses a multi-head attention mechanism for the combination of structure aware embedding from the Crystal Graph Convolution Network (CGCNN) and text embeddings from the SciBERT model. We train our model in an end-to-end framework using data from the Materials Project Dataset. We show that our proposed model shows an improvement compared to the vanilla CGCNN and SciBERT model for all four key properties: formation energy, band gap, energy above hull and fermi energy. Specifically, we observe an improvement of 40% compared to the vanilla CGCNN model and 68% compared to the SciBERT model for predicting the formation energy per atom. Importantly, we demonstrate the zero shot performance of the trained model on small curated datasets of Perovskites, Chalcogenides and the Jarvis Dataset. The results show that the proposed model exhibits better zero shot performance than the individual plain vanilla CGCNN and SciBERT model. This enables researchers to deploy the model for specialized industrial applications where collection of training data is prohibitively expensive.

Figures (9)

• Figure 1: The figure provides an overview of MatMMFuse. The CGCNN model generates a structure aware embedding while the SciBERT model generates a context aware embedding which are combined using a multi-head attention mechanism.
• Figure 2: The scatter plot presents the actual versus predicted values for (\ref{['fig:image1']}) Formation Energy per atom, (\ref{['fig:image2']}) Band Gap. The model tends to incorrectly predict the values for band gap when the values are close to zero.
• Figure 3: The t-SNE plot of the embedding from the embedding for the test dataset from Materials Project, with each point representing an individual crystal. Colors for each point are associated with their formation energies. (\ref{['fig:tsne1']}) CGCNN embedding, (\ref{['fig:tsne2']}) SciBERT embedding , (\ref{['fig:tsne3']}) MatMMFuse embedding
• Figure 4: The scatter plot presents a comparison between the actual and the predicted values of formation energy per atom for the JARVIS dataset.
• Figure 5: The plot compares the performance of different BERT models for encoding the text representation. MatSciBERT has the best performance and ALBERT has the worst performance.