Hierarchical Network Fusion for Multi-Modal Electron Micrograph Representation Learning with Foundational Large Language Models

TL;DR

This work introduces MultiFusion-LLM, a multimodal framework for nanomaterial identification in electron micrographs that fuses patch-sequence representations with vision-graph priors across multiple patch scales via Hierarchical Network Fusion (HNF). It further enhances domain knowledge by generating detailed nanomaterial descriptions with zero-shot Chain-of-Thought prompting from large language models and pretraining small LMs on these texts, followed by cross-modal fusion through multi-head attention. The method achieves state-of-the-art performance on the SEM dataset aversa2018first, with significant improvements over strong vision and graph baselines and robust ablations validating the contribution of HNF, LLM-derived descriptions, and cross-modal attention. The approach promises improved robustness to distributional shifts and supports high-throughput screening in semiconductor materials research by integrating image-based representations with linguistic domain knowledge.

Abstract

Characterizing materials with electron micrographs is a crucial task in fields such as semiconductors and quantum materials. The complex hierarchical structure of micrographs often poses challenges for traditional classification methods. In this study, we propose an innovative backbone architecture for analyzing electron micrographs. We create multi-modal representations of the micrographs by tokenizing them into patch sequences and, additionally, representing them as vision graphs, commonly referred to as patch attributed graphs. We introduce the Hierarchical Network Fusion (HNF), a multi-layered network structure architecture that facilitates information exchange between the multi-modal representations and knowledge integration across different patch resolutions. Furthermore, we leverage large language models (LLMs) to generate detailed technical descriptions of nanomaterials as auxiliary information to assist in the downstream task. We utilize a cross-modal attention mechanism for knowledge fusion across cross-domain representations(both image-based and linguistic insights) to predict the nanomaterial category. This multi-faceted approach promises a more comprehensive and accurate representation and classification of micrographs for nanomaterial identification. Our framework outperforms traditional methods, overcoming challenges posed by distributional shifts, and facilitating high-throughput screening.

Figures (9)

• Figure 1: The figure provides a visual representation of the challenges of classifying electron micrographs in the SEM dataset(aversa2018first).
• Figure 2: Our framework includes three methods: (a) Hierarchical Network Fusion (HNF), (b) Zero-shot Chain-of-Thought (Zero-Shot CoT) prompting with large language models (LLMs), and (c) an output layer modeled with the multi-head attention (MHA) mechanism vaswani2017attention for integrating cross-domain embeddings and facilitating label prediction. LLMs take a prompt, not an electron micrograph, as input.
• Figure 3: Overview of the HNF module. The HNF module utilizes a multi-layered network with increasing patch sizes to represent the electron micrograph-based patch sequence and vision graph at various scales, facilitating computation of hierarchical embeddings that encapsulate the global context. The cascaded structure incorporates multiple stacked layers; each layer involves bidirectional Neural ODEs and Graph Chebyshev convolution to compute patch sequence and vision graph embeddings, respectively. A gating mechanism integrates these cross-domain embeddings, generating unified hierarchical embeddings that offer a comprehensive view of the electron micrographs. Overall, the HNF module, facilitates seamless information fusion at multiple scales, producing a cohesive representation of the micrographs. $<\space\textit{cls}\space>$ is the cls token and VN is the virtual node. $h^{l}_{i}$ and $e^{l}_{i}$ denotes the patch and node representation at layer $l$ of patch or node $i$, respectively.
• Figure 4: The figure depicts the different types of nanomaterials found in the SEM dataset (aversa2018first) (left to right in the first row: biological, fibers, films, MEMS, nanowires; left to right in the second row: particles, patterned surface, porous sponges, powder, tips).
• Figure 5: Overall, the architecture of our framework involves using zero-shot CoT prompting with LLMs to generate technical textual descriptions and pre-train smaller language models (LMs) using masked language modeling (MLM). We then jointly optimize the smaller LM along with the HNF method in supervised learning tasks, aiming to minimize cross-entropy loss and improve multi-class classification accuracy.