Vision HgNN: An Electron-Micrograph is Worth Hypergraph of Hypernodes

TL;DR

This work tackles the challenge of classifying electron micrographs, which exhibit complex, multi-scale patterns and long-tailed category distributions. It introduces Vision-HgNN, a hypergraph neural network that learns visual hypergraphs from image patches via HgSL and performs relational inference with HgAT and HgT, followed by HgRo readout to predict material categories. The approach achieves state-of-the-art performance on SEM datasets, with strong ablation support showing the complementary benefits of local/global hyperedge-node attention and long-range dependencies, while remaining efficient on large image collections. The framework promises broader applicability across electron microscopy modalities and downstream material-characterization tasks such as anomaly detection and segmentation.

Abstract

Material characterization using electron micrographs is a crucial but challenging task with applications in various fields, such as semiconductors, quantum materials, batteries, etc. The challenges in categorizing electron micrographs include but are not limited to the complexity of patterns, high level of detail, and imbalanced data distribution(long-tail distribution). Existing methods have difficulty in modeling the complex relational structure in electron micrographs, hindering their ability to effectively capture the complex relationships between different spatial regions of micrographs. We propose a hypergraph neural network(HgNN) backbone architecture, a conceptually alternative approach, to better model the complex relationships in electron micrographs and improve material characterization accuracy. By utilizing cost-effective GPU hardware, our proposed framework outperforms popular baselines. The results of the ablation studies demonstrate that the proposed framework is effective in achieving state-of-the-art performance on benchmark datasets and efficient in terms of computational and memory requirements for handling large-scale electron micrograph-based datasets.

Figures (8)

• Figure 1: The figure depicts the various challenges in the electron micrograph classification task on the SEM dataset(aversa2018first).
• Figure 2: For illustration purpose, an electron micrograph(MEMS device, aversa2018first) was split into $\text{3} \times \text{3}$ patches. This figure depicts a regular grid, a sequence, a graph, and a hypergraph representation of an electron micrograph. (a) ConvNets operate on a grid of pixels, (b) ViTs operate on a sequence of grid-like patches, (c) GNNs operate on visual graphs where patches are viewed as nodes, and (d) HgNNs operate on visual hypergraphs where the patches represent the hypernodes to perform classification tasks. The visual graph and hypergraph structure representations are learned through the nearest neighbor search algorithm. They are linked based on the visual content and are not necessarily determined by their spatial location in the micrograph. The edges in the graph model pair-wise relations among the patches, while hyperedges model multi-dyadic relationships.
• Figure 3: The isotropic $\text{Vision-HgNN}$ architecture. $y^{p}_{i}$ denotes the model predictions.
• Figure 4: For illustration, we split the electron micrograph into $3 \times 3$ patches. We represent the electron micrograph as a patch-attributed visual hypergraph. The framework presents an end-to-end visual hypergraph representation learning with Hypergraph Neural Networks for categorization tasks.
• Figure 5: The figure shows the electron micrograph categories in the SEM dataset (aversa2018first)(left to right in the first row: biological, fibers, films, MEMS, nanowires; left to right in the second row: particles, patterned surface, porous sponges, powder, tips).