SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels

TL;DR

This work tackles handwritten mathematical expression recognition with limited labeled data by introducing SemiHMER, a dual-branch cross-head framework that leverages pseudo-labels from one head to supervise the other under a weak-to-strong augmentation regime. It couples this with a Global Dynamic Counting Module to refine decoding in long-distance formulas and repeated-symbol scenarios. The method demonstrates substantial gains on CROHME benchmarks, reporting average improvements around 5 percentage points over baselines, validating the effectiveness of combining consistency regularization, pseudo-supervision, and counting-based guidance for HMER. Overall, SemiHMER advances semi-supervised learning for structured formula recognition and highlights strong practical impact for reducing labeling costs while boosting accuracy.

Abstract

In this paper, we study semi-supervised Handwritten Mathematical Expression Recognition (HMER) via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization framework, termed SemiHMER, which introduces dual-branch semi-supervised learning. Specifically, we enforce consistency between the two networks for the same input image. The pseudo-label, generated by one perturbed recognition network, is utilized to supervise the other network using the standard cross-entropy loss. The SemiHMER consistency encourages high similarity between the predictions of the two perturbed networks for the same input image and expands the training data by leveraging unlabeled data with pseudo-labels. We further introduce a weak-to-strong strategy by applying different levels of augmentation to each branch, effectively expanding the training data and enhancing the quality of network training. Additionally, we propose a novel module, the Global Dynamic Counting Module (GDCM), to enhance the performance of the HMER decoder by alleviating recognition inaccuracies in long-distance formula recognition and reducing the occurrence of repeated characters. The experimental results demonstrate that our work achieves significant performance improvements, with an average accuracy increase of 5.47% on CROHME14, 4.87% on CROHME16, and 5.25% on CROHME19, compared to our baselines.

Figures (3)

• Figure 1: Overview of our Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels. In this scheme, we use ${Decoder_1}$ as an example. Weakly augmented (non-augmented) labeled data flows through the encoder module and the corresponding prediction ${Decoder_1}$ to produce the prediction ${Classification\ Vector_{super,weak}}$, which is then supervised by the ground truth. At the same time, the prediction ${Classification\ Vector_{super,weak}}$ will be transformed into hard pseudo labels, which also utilized as a supervision signal for the strongly augmented image prediction ${Classification\ Vector_{super,strong}}$ from the other head. The ${Classification\ Vector_{super,strong}}$ is also supervised by the ground truth labels. Besides that, We use predictions ${Classification\ Vector_{unsuper,weak}}$ from weakly augmented unlabeled data as ground truth to supervise the encoder and predictions ${Classification\ Vector_{unsuper,strong}}$ on strongly augmented data. Augmentation alternation refers to alternately switching the intensity of augmentation applied to different decoders.
• Figure 2: Structure of the proposed decoder based on the Global Dynamic Counting Module (GDCM)
• Figure 3: Some example images from the CROHME dataset.