MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science

TL;DR

A comprehensive evaluation convincingly supports the strength of MELT, demonstrating superior performance compared to existing continued pre-training methods and highlighting its broad applicability across a wide spectrum of materials science.

Abstract

We introduce a novel continued pre-training method, MELT (MatEriaLs-aware continued pre-Training), specifically designed to efficiently adapt the pre-trained language models (PLMs) for materials science. Unlike previous adaptation strategies that solely focus on constructing domain-specific corpus, MELT comprehensively considers both the corpus and the training strategy, given that materials science corpus has distinct characteristics from other domains. To this end, we first construct a comprehensive materials knowledge base from the scientific corpus by building semantic graphs. Leveraging this extracted knowledge, we integrate a curriculum into the adaptation process that begins with familiar and generalized concepts and progressively moves toward more specialized terms. We conduct extensive experiments across diverse benchmarks to verify the effectiveness and generality of MELT. A comprehensive evaluation convincingly supports the strength of MELT, demonstrating superior performance compared to existing continued pre-training methods. The in-depth analysis also shows that MELT enables PLMs to effectively represent materials entities compared to the existing adaptation methods, thereby highlighting its broad applicability across a wide spectrum of materials science.

Figures (8)

• Figure 1: Frequency histograms of all words and chemical formulas on materials science corpus (150K materials-related scientific papers).
• Figure 2: Overall adaptation process of PLMs with Melt. Starting from the materials science corpus, we extract chemical entities (e.g., chemical names and formulas). These entities are then expanded to related terms based on the materials embeddings with their compositional property, resulting in materials semantic graphs. Based on the constructed graphs, Melt performs curriculum masking over materials corpus (e.g., three phases in this example).
• Figure 3: Masking ratio (%) of the materials entities in token classification datasets (NER, RC, SF).
• Figure 4: Categorical accuracy (%) for the classes of the chemical entities in the SF task.
• Figure 5: Comparison of the Macro F1 score for the SOFC-NER and SOFC-Filling test sets over the different number of pre-training steps.