Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CER-HV: A CER-Based Human-in-the-Loop Framework for Cleaning Datasets Applied to Arabic-Script HTR

Sana Al-azzawi, Elisa Barney, Marcus Liwicki

TL;DR

This work shows that data quality significantly limits Arabic-script handwritten text recognition. It introduces CER-HV, a two-stage framework that replaces loss-based noise scoring with CER-based ranking and couples automatic detection with a HITL verification step to identify and clean label errors. The authors demonstrate state-of-the-art CRNN performance on six datasets without data cleaning and quantify substantial improvements when noisy labels are corrected or removed, especially in noisier datasets. They provide cleaned evaluation splits and error annotations to enable reproducible benchmarking, and discuss the broader implications of data quality for fair model comparison and future research in HTR. Collectively, CER-HV emphasizes that robust benchmarks require reliable labels as much as strong models, and offers a practical pathway for dataset validation and active learning in resource-constrained handwriting datasets.

Abstract

Handwritten text recognition (HTR) for Arabic-script languages still lags behind Latin-script HTR, despite recent advances in model architectures, datasets, and benchmarks. We show that data quality is a significant limiting factor in many published datasets and propose CER-HV (CER-based Ranking with Human Verification) as a framework to detect and clean label errors. CER-HV combines a CER-based noise detector, built on a carefully configured Convolutional Recurrent Neural Network (CRNN) with early stopping to avoid overfitting noisy samples, and a human-in-the-loop (HITL) step that verifies high-ranking samples. The framework reveals that several existing datasets contain previously underreported problems, including transcription, segmentation, orientation, and non-text content errors. These have been identified with up to 90 percent precision in the Muharaf and 80-86 percent in the PHTI datasets. We also show that our CRNN achieves state-of-the-art performance across five of the six evaluated datasets, reaching 8.45 percent Character Error Rate (CER) on KHATT (Arabic), 8.26 percent on PHTI (Pashto), 10.66 percent on Ajami, and 10.11 percent on Muharaf (Arabic), all without any data cleaning. We establish a new baseline of 11.3 percent CER on the PHTD (Persian) dataset. Applying CER-HV improves the evaluation CER by 0.3-0.6 percent on the cleaner datasets and 1.0-1.8 percent on the noisier ones. Although our experiments focus on documents written in an Arabic-script language, including Arabic, Persian, Urdu, Ajami, and Pashto, the framework is general and can be applied to other text recognition datasets.

CER-HV: A CER-Based Human-in-the-Loop Framework for Cleaning Datasets Applied to Arabic-Script HTR

TL;DR

This work shows that data quality significantly limits Arabic-script handwritten text recognition. It introduces CER-HV, a two-stage framework that replaces loss-based noise scoring with CER-based ranking and couples automatic detection with a HITL verification step to identify and clean label errors. The authors demonstrate state-of-the-art CRNN performance on six datasets without data cleaning and quantify substantial improvements when noisy labels are corrected or removed, especially in noisier datasets. They provide cleaned evaluation splits and error annotations to enable reproducible benchmarking, and discuss the broader implications of data quality for fair model comparison and future research in HTR. Collectively, CER-HV emphasizes that robust benchmarks require reliable labels as much as strong models, and offers a practical pathway for dataset validation and active learning in resource-constrained handwriting datasets.

Abstract

Handwritten text recognition (HTR) for Arabic-script languages still lags behind Latin-script HTR, despite recent advances in model architectures, datasets, and benchmarks. We show that data quality is a significant limiting factor in many published datasets and propose CER-HV (CER-based Ranking with Human Verification) as a framework to detect and clean label errors. CER-HV combines a CER-based noise detector, built on a carefully configured Convolutional Recurrent Neural Network (CRNN) with early stopping to avoid overfitting noisy samples, and a human-in-the-loop (HITL) step that verifies high-ranking samples. The framework reveals that several existing datasets contain previously underreported problems, including transcription, segmentation, orientation, and non-text content errors. These have been identified with up to 90 percent precision in the Muharaf and 80-86 percent in the PHTI datasets. We also show that our CRNN achieves state-of-the-art performance across five of the six evaluated datasets, reaching 8.45 percent Character Error Rate (CER) on KHATT (Arabic), 8.26 percent on PHTI (Pashto), 10.66 percent on Ajami, and 10.11 percent on Muharaf (Arabic), all without any data cleaning. We establish a new baseline of 11.3 percent CER on the PHTD (Persian) dataset. Applying CER-HV improves the evaluation CER by 0.3-0.6 percent on the cleaner datasets and 1.0-1.8 percent on the noisier ones. Although our experiments focus on documents written in an Arabic-script language, including Arabic, Persian, Urdu, Ajami, and Pashto, the framework is general and can be applied to other text recognition datasets.
Paper Structure (20 sections, 7 equations, 6 figures, 5 tables, 1 algorithm)

This paper contains 20 sections, 7 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Arabic-script HTR
  4. Label Error Detection in Handwritten Text Recognition
  5. Method
  6. Problem Formulation
  7. HTR Model: CRNN Architecture
  8. CER-HV Framework: CER Based Ranking and HITL Verification
  9. Experimental Setup
  10. Datasets
  11. Training Setup and Evaluation
  12. Training setup
  13. Evaluation Metric
  14. Results
  15. Quantitative Benchmarking of the CRNN-Based HTR System
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Representative examples of error categories found in the datasets.
  • Figure 2: Overview of the CER-HV framework. The first stage employs the learning dynamics of the CRNN and scores each sample using CER, and the second stage applies a human verification step to the highest ranking samples ($CER > \tau$).
  • Figure 3: The CRNN architecture used as the base recognition model in Stage 1 of the CER-HV framework. The model combines a convolutional encoder for visual feature extraction with an RNN decoder and a CTC output layer for sequence prediction.
  • Figure 4: Examples of handwriting from six datasets. From top to bottom: KHATT (Arabic), Muharaf (Arabic), PHTI (Pashto), PHTD (Persian), NUST-UHWR (Urdu), Ajami Hausa, and Ajami Fulfulde.
  • Figure 5: Representative examples of flagged samples from the major error categories identified through the CER-HV pipeline: (a) transcription error, (b) segmentation error, (c) orientation error involving a 180° rotation, (d) orientation error involving diagonally oriented text, where a red arrow indicates the intended reading direction, (e) script mismatch, and (f) irrelevant or non-text content. For transcription errors (a), the model prediction is shown to illustrate the mismatch with the ground truth. For structural errors where correction is straightforward (b–c), the corrected image is displayed. Dataset name, image ID, CER, and ground-truth text are shown for all samples. Examples are drawn from multiple datasets (Muharaf, NUST-UHWR, PHTI, and Ajami), highlighting the diverse manifestations of labeling challenges across Arabic-script handwriting corpora.
  • ...and 1 more figures