Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition

Jianhua Zhu, Liangcai Gao, Wenqi Zhao

TL;DR

By modeling and utilizing implicit character information, ICAL achieves a more accurate and context-aware interpretation of handwritten mathematical expressions and notably surpasses the state-of-the-art SOTA models.

Abstract

Significant progress has been made in the field of handwritten mathematical expression recognition, while existing encoder-decoder methods are usually difficult to model global information in $LaTeX$. Therefore, this paper introduces a novel approach, Implicit Character-Aided Learning (ICAL), to mine the global expression information and enhance handwritten mathematical expression recognition. Specifically, we propose the Implicit Character Construction Module (ICCM) to predict implicit character sequences and use a Fusion Module to merge the outputs of the ICCM and the decoder, thereby producing corrected predictions. By modeling and utilizing implicit character information, ICAL achieves a more accurate and context-aware interpretation of handwritten mathematical expressions. Experimental results demonstrate that ICAL notably surpasses the state-of-the-art(SOTA) models, improving the expression recognition rate (ExpRate) by 2.25\%/1.81\%/1.39\% on the CROHME 2014/2016/2019 datasets respectively, and achieves a remarkable 69.06\% on the challenging HME100k test set. We make our code available on the GitHub: https://github.com/qingzhenduyu/ICAL

ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition

TL;DR

By modeling and utilizing implicit character information, ICAL achieves a more accurate and context-aware interpretation of handwritten mathematical expressions and notably surpasses the state-of-the-art SOTA models.

Abstract

Significant progress has been made in the field of handwritten mathematical expression recognition, while existing encoder-decoder methods are usually difficult to model global information in . Therefore, this paper introduces a novel approach, Implicit Character-Aided Learning (ICAL), to mine the global expression information and enhance handwritten mathematical expression recognition. Specifically, we propose the Implicit Character Construction Module (ICCM) to predict implicit character sequences and use a Fusion Module to merge the outputs of the ICCM and the decoder, thereby producing corrected predictions. By modeling and utilizing implicit character information, ICAL achieves a more accurate and context-aware interpretation of handwritten mathematical expressions. Experimental results demonstrate that ICAL notably surpasses the state-of-the-art(SOTA) models, improving the expression recognition rate (ExpRate) by 2.25\%/1.81\%/1.39\% on the CROHME 2014/2016/2019 datasets respectively, and achieves a remarkable 69.06\% on the challenging HME100k test set. We make our code available on the GitHub: https://github.com/qingzhenduyu/ICAL
Paper Structure (19 sections, 13 equations, 3 figures, 5 tables)

This paper contains 19 sections, 13 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Methods
  4. Deep Learning Methods
  5. Methodology
  6. Visual Encoder
  7. Transformer Decoder with ARM
  8. Implicit Character Construction Module
  9. Fusion Module
  10. Loss Function
  11. Experiments
  12. Dataset
  13. Evaluation Metrics
  14. Implementation Details
  15. Comparison with State-of-the-art Methods
  16. ...and 4 more sections

Figures (3)

  • Figure 1: (a)Illustration of past method which uses DenseNet Encoder and RNN/Transformer Decoder. (b) Our ICAL method aided by implicit character learning. The characters highlighted in red signify inaccuracies in the prediction, whereas the blue highlights denote implicit characters.
  • Figure 2: The architecture of ICAL model (left) and Coverage Attention (right). To simplify the illustration, we have condensed the depiction of bidirectional training in the figure.
  • Figure 3: Case studies for the Ground Truth and CoMER, ICAL methods. The red symbols represent incorrect predictions. 'ICCM' represents the implicit character sequence predicted by the ICCM module, where $\texttt{<s>}$ is the abbreviation for the $\texttt{<space>}$ token.