Symbol-Aware Reasoning with Masked Discrete Diffusion for Handwritten Mathematical Expression Recognition

TL;DR

This work reframes handwritten mathematical expression recognition (HMER) as a discrete diffusion process that iteratively refines symbols and two-dimensional structure, rather than generating LaTeX tokens autoregressively. It introduces Symbol-Aware Tokenization (SAT) to align visible symbols with per-symbol modifiers and Random-Masking Mutual Learning (RMML) to enforce cross-view consistency under masking, together yielding robust structural reasoning. Empirical results on MathWriting and CROHME show state-of-the-art performance, with a MathWriting CER of $5.56\%$ and EM of $60.42\%$, and consistent improvements across CROHME editions, including a 2023 EM of $60.78\%$. The approach demonstrates a new paradigm for structure-aware recognition beyond generative models, balancing accuracy, stability, and efficiency in offline HMER.

Abstract

Handwritten Mathematical Expression Recognition (HMER) requires reasoning over diverse symbols and 2D structural layouts, yet autoregressive models struggle with exposure bias and syntactic inconsistency. We present a discrete diffusion framework that reformulates HMER as iterative symbolic refinement instead of sequential generation. Through multi-step remasking, the proposal progressively refines both symbols and structural relations, removing causal dependencies and improving structural consistency. A symbol-aware tokenization and Random-Masking Mutual Learning further enhance syntactic alignment and robustness to handwriting diversity. On the MathWriting benchmark, the proposal achieves 5.56\% CER and 60.42\% EM, outperforming strong Transformer and commercial baselines. Consistent gains on CROHME 2014--2023 demonstrate that discrete diffusion provides a new paradigm for structure-aware visual recognition beyond generative modeling.

Figures (7)

• Figure 1: Illustration of the symbolic reasoning process in the proposal. (a) Handwritten expression. (b) Symbol refinement. (c) Modifier refinement. Starting from a fully masked sequence, the proposal iteratively unmasks and re-masks tokens through discrete diffusion to jointly recover symbols and structural relations.
• Figure 2: Overall architecture of the proposal. The framework comprises a Vision Transformer (ViT) image encoder and a formula decoder. The encoder extracts visual features from a $224 \times 224$ handwritten expression image, while the decoder iteratively refines masked LaTeX tokens through discrete diffusion to reconstruct the complete expression.
• Figure 3: Example of structural editing in a LaTeX token sequence. When removing a superscript (e.g., converting x^2 to x2), multiple dependent tokens must be deleted and subsequent token positions shift forward, altering positional encodings. This demonstrates that directly applying diffusion to raw LaTeX sequences can easily break syntactic consistency, motivating the need for a symbol-level formulation.
• Figure 4: Illustration of symbol-aware tokenization. A LaTeX expression is decomposed into visible symbols and structural modifiers (e.g., braces, superscripts, subscripts), which are aligned on a one-to-one basis. By integrating modifier embeddings into the corresponding symbols, the model forms a unified symbol-level representation that preserves syntactic consistency while simplifying grammar. This representation shortens sequence length and stabilizes diffusion-based reconstruction.
• Figure 5: Overview of Random-Masking Mutual Learning (RMML). Two randomly masked variants of the same formula are reconstructed through a shared diffusion decoder, and their predicted token distributions are aligned via KL-divergence regularization to enhance robustness to handwriting and structural variations.