Link prediction Graph Neural Networks for structure recognition of Handwritten Mathematical Expressions

TL;DR

This work targets online handwritten mathematical expression recognition by modeling HMEs as symbol-level graphs, bypassing dense stroke graphs. It combines a deep BLSTM for symbol segmentation/recognition and initial relations with a 2D-CYK parser to enumerate plausible relations, followed by an Edge Graph Attention Network to perform edge pruning and produce a Symbol Layout Tree. Evaluations on the CROHME benchmarks show that CYK parsing with LOS filtering balances graph completeness and sparsity, while EGAT-based link prediction yields reliable edge decisions and competitive expression-level accuracy. The approach offers an interpretable, explicit graph representation that could facilitate editing and manipulation, with potential for real-time applications and integration with more robust symbol recognizers.

Abstract

We propose a Graph Neural Network (GNN)-based approach for Handwritten Mathematical Expression (HME) recognition by modeling HMEs as graphs, where nodes represent symbols and edges capture spatial dependencies. A deep BLSTM network is used for symbol segmentation, recognition, and spatial relation classification, forming an initial primitive graph. A 2D-CFG parser then generates all possible spatial relations, while the GNN-based link prediction model refines the structure by removing unnecessary connections, ultimately forming the Symbol Label Graph. Experimental results demonstrate the effectiveness of our approach, showing promising performance in HME structure recognition.

Figures (3)

• Figure 1: Overview of the method
• Figure 2: Symbol‐level graph constructed by deep BLSTM and 2D CYK parsing.
• Figure 3: The link prediction EGAT refines the initial symbol‐level graph, removing edges to form an SLT.