Preliminary Investigations of a Multi-Faceted Robust and Synergistic Approach in Semiconductor Electron Micrograph Analysis: Integrating Vision Transformers with Large Language and Multimodal Models

TL;DR

Automated nanomaterial identification from SEM micrographs is challenging due to intra-class variability and inter-class similarity. The authors propose CM-EMRL, a cross-modal pipeline that fuses a ViT-based image encoder with domain knowledge from large language models via zero-shot CoT prompting and few-shot prompts from large multimodal models, all integrated through a unified attention layer. The approach yields state-of-the-art results on SEM datasets and generalizes to additional benchmarks, with ablations confirming the value of each component (LLM prompts, LMM prompts, and cross-modal fusion). This work offers a scalable, interpretable framework that blends linguistic and visual signals to enable robust high-throughput nanomaterial screening for semiconductor manufacturing.

Abstract

Characterizing materials using electron micrographs is crucial in areas such as semiconductors and quantum materials. Traditional classification methods falter due to the intricatestructures of these micrographs. This study introduces an innovative architecture that leverages the generative capabilities of zero-shot prompting in Large Language Models (LLMs) such as GPT-4(language only), the predictive ability of few-shot (in-context) learning in Large Multimodal Models (LMMs) such as GPT-4(V)ision, and fuses knowledge across image based and linguistic insights for accurate nanomaterial category prediction. This comprehensive approach aims to provide a robust solution for the automated nanomaterial identification task in semiconductor manufacturing, blending performance, efficiency, and interpretability. Our method surpasses conventional approaches, offering precise nanomaterial identification and facilitating high-throughput screening.

Figures (10)

• Figure 1: The figure provides a visual representation of the challenges of classifying electron micrographs in the SEM datasetaversa2018first.
• Figure 2: The electron micrographs shown above were provided as input to GPT-4Vyang2023dawn for nanomaterial categorization to determine how the multimodal model classifies nanomaterials in SEM images across different structural categories from a predefined list. However, the LMMs predictions were incorrect, with the actual nanomaterial categories being films, nanowires, MEMS and powder. It highlights the inherent limitations of visual processing capabilities of even advanced LMMs such as GPT-4V, reminding users to approach predictions with a degree of skepticism.
• Figure 3: Our framework includes three methods: (a) Image Encoder (ViT), (b) Zero-Shot CoT prompting with LLMs, (c) Few-shot prompting with LMMs, and (d) an output layer modeled with the multi-head attention (MHA) mechanism for integrating cross-domain embeddings and facilitating label prediction.
• Figure 4: The figure showcases nanomaterials from the SEM datasetaversa2018first. In the first, second and third rows (from left to right), we have: biological, fibers, films, MEMS, nanowires, particles, and patterned surface, porous sponges, powder, respectively. Meanwhile, the last row displays: tips.
• Figure 5: The figure compares our proposed framework to vision-based supervised convolutional neural networks (ConvNets), vision transformers (ViTs), and self-supervised learning (VSL) algorithms on the SEM dataset aversa2018first.