Element2Vec: Build Chemical Element Representation from Text for Property Prediction

TL;DR

This work introduces Element2Vec, a language-informed framework that converts textual descriptions of chemical elements into both Global and Local embeddings using a Gemini-based LLM. By labeling eight attribute categories and augmenting attribute-local texts with a concise element-wide summary, Element2Vec captures diverse facet signals while preserving global context. The authors show that Local, summary-augmented embeddings yield tighter element clusters and more decisive predictions, and they address data sparsity through a test-time self-attention training approach that effectively imputes missing property values. Collectively, the approach bridges textual knowledge and physical properties to support interpretable, scalable materials discovery, with demonstrated gains in element-family classification and atomic-property prediction such as the van der Waals radius."

Abstract

Accurate property data for chemical elements is crucial for materials design and manufacturing, but many of them are difficult to measure directly due to equipment constraints. While traditional methods use the properties of other elements or related properties for prediction via numerical analyses, they often fail to model complex relationships. After all, not all characteristics can be represented as scalars. Recent efforts have been made to explore advanced AI tools such as language models for property estimation, but they still suffer from hallucinations and a lack of interpretability. In this paper, we investigate Element2Vecto effectively represent chemical elements from natural languages to support research in the natural sciences. Given the text parsed from Wikipedia pages, we use language models to generate both a single general-purpose embedding (Global) and a set of attribute-highlighted vectors (Local). Despite the complicated relationship across elements, the computational challenges also exist because of 1) the discrepancy in text distribution between common descriptions and specialized scientific texts, and 2) the extremely limited data, i.e., with only 118 known elements, data for specific properties is often highly sparse and incomplete. Thus, we also design a test-time training method based on self-attention to mitigate the prediction error caused by Vanilla regression clearly. We hope this work could pave the way for advancing AI-driven discovery in materials science.

Figures (9)

• Figure 1: Encode Element Representation from Text based on inspiration from Image Representation. (Left) Conventional image representation can be either a global embedding extracted from the whole image, or a set of local embeddings extracted from a sub-region that focuses on several key components ma2024mode, such as tail and head. (Right) Analogously, given the tremendous amount of text in element description and its diverse application and aspects, we can also extract a single Global embedding from all text or following the definition in cardarelli2008materials to extract Local embeddings highlighting different element attributes like atomic, chemical, thermal, and optical.
• Figure 1: (a) The LR result of all Element2Vec-Locals embedding datasets with full attributes vector. (B) The LR result of all Element2Vec-Locals embedding datasets with only ARF attribute vector.
• Figure 2: Element2Vec-Locals Embeddings Extraction. For each element, given the text parsed from the Wikipedia webpage, we use LLM to label each sentence about the attribute categories cardarelli2008materials independently and summarize the full text to shorten the words via the prompts $\mathbf{p}_c$ and $\mathbf{p}_g$ respectively. For each subset, it is concatenated with the summary and fed into $f_{emb}$ for extraction.
• Figure 3: t-SNE visualization of element2vec representations. (a) Embeddings obtained by sending the entire Wikipedia page with a prompt. (b) Local embeddings obtained from 8 attributes without summary. (c) Local embedding, one point per element obtained from the attribute plus its summary. (d) Aggregated embeddings from (c), 8 attribute-level embeddings are combined into a single point per element. (e) Global embedding: one point per element obtained directly. (f) Periodic table with the same color legend as in (a–e).
• Figure 4: Entropy distributions for family classification. Kernel density estimation (KDE) of prediction entropy for the 10-way periodic-family classifier under three embeddings.