Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reading Order Independent Metrics for Information Extraction in Handwritten Documents

David Villanova-Aparisi, Solène Tarride, Carlos-D. Martínez-Hinarejos, Verónica Romero, Christopher Kermorvant, Moisés Pastor-Gadea

TL;DR

This work tackles the problem that traditional information-extraction evaluation on handwritten documents is biased by reading order. It introduces reading-order independent metrics, including OIECER, OIEWER, OINerval, and bag-of-words/entity variants, solved via assignment-based matching and the Hungarian algorithm to remove sequence constraints. The authors validate these metrics with a DAN-based HTR+NER model across IAM, Simara, Esposalles, POPP, and a real-world French Military Records corpus, showing that OI metrics correlate with standard ones on regular data yet better reflect real-world unordered annotations. They recommend ECER and EWER for general benchmarking, reserve Nerval for CER-tolerance scenarios, and provide an open-source Python package to compute the metrics, enabling fair, cross-dataset evaluation for handwritten IE systems.

Abstract

Information Extraction processes in handwritten documents tend to rely on obtaining an automatic transcription and performing Named Entity Recognition (NER) over such transcription. For this reason, in publicly available datasets, the performance of the systems is usually evaluated with metrics particular to each dataset. Moreover, most of the metrics employed are sensitive to reading order errors. Therefore, they do not reflect the expected final application of the system and introduce biases in more complex documents. In this paper, we propose and publicly release a set of reading order independent metrics tailored to Information Extraction evaluation in handwritten documents. In our experimentation, we perform an in-depth analysis of the behavior of the metrics to recommend what we consider to be the minimal set of metrics to evaluate a task correctly.

Reading Order Independent Metrics for Information Extraction in Handwritten Documents

TL;DR

This work tackles the problem that traditional information-extraction evaluation on handwritten documents is biased by reading order. It introduces reading-order independent metrics, including OIECER, OIEWER, OINerval, and bag-of-words/entity variants, solved via assignment-based matching and the Hungarian algorithm to remove sequence constraints. The authors validate these metrics with a DAN-based HTR+NER model across IAM, Simara, Esposalles, POPP, and a real-world French Military Records corpus, showing that OI metrics correlate with standard ones on regular data yet better reflect real-world unordered annotations. They recommend ECER and EWER for general benchmarking, reserve Nerval for CER-tolerance scenarios, and provide an open-source Python package to compute the metrics, enabling fair, cross-dataset evaluation for handwritten IE systems.

Abstract

Information Extraction processes in handwritten documents tend to rely on obtaining an automatic transcription and performing Named Entity Recognition (NER) over such transcription. For this reason, in publicly available datasets, the performance of the systems is usually evaluated with metrics particular to each dataset. Moreover, most of the metrics employed are sensitive to reading order errors. Therefore, they do not reflect the expected final application of the system and introduce biases in more complex documents. In this paper, we propose and publicly release a set of reading order independent metrics tailored to Information Extraction evaluation in handwritten documents. In our experimentation, we perform an in-depth analysis of the behavior of the metrics to recommend what we consider to be the minimal set of metrics to evaluate a task correctly.
Paper Structure (38 sections, 11 equations, 3 figures, 7 tables)

This paper contains 38 sections, 11 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Metrics Based On Word Position Alignment
  4. Metrics Based On Text Alignment
  5. Alignment-Free Metrics
  6. Discussion
  7. Evaluated Metrics
  8. Order Dependent Metrics
  9. Character Error Rate (CER) and Word Error Rate (WER)
  10. Entity Character Error Rate (ECER) and Entity Word Error Rate (EWER)
  11. Soft-aligned Entity Precision, Recall and F1-Score (Nerval)
  12. Assignment-based Proposed Reading Order Independent Metrics
  13. Reading Order Independent Entity Character Error Rate (OIECER) and Entity Word Error Rate (OIEWER)
  14. Reading Order Independent Soft-aligned Entity Precision, Recall and F1-Score (OINerval)
  15. Bag-of-words-based Reading Order Independent Metrics
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Illustration of the five datasets used in this study. Named Entities are highlighted. Not all entity types appear in these examples.
  • Figure 2: Absolute linear correlation (Pearson) between the different metrics across the four datasets. The correlation value appears on each cell, as well as an indication of its p-value: * indicates a $p-value < 0.05$, ** indicates a $p-value < 0.01$, and *** indicates a $p-value < 0.001$. No star indicates that the correlation is not significant.
  • Figure 3: Absolute rank correlation (Spearman) between the different metrics across the four datasets. The correlation value appears on each cell, as well as an indication of its p-value: * indicates a $p-value < 0.05$, ** indicates a $p-value < 0.01$, and *** indicates a $p-value < 0.001$. No star indicates that the correlation is not significant.