Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

jp-evalb: Robust Alignment-based PARSEVAL Measures

Jungyeul Park, Junrui Wang, Eunkyul Leah Jo, Angela Yoonseo Park

TL;DR

This paper addresses limitations of the traditional PARSEVAL evaluation (evalb) which assumes exact tokenization and sentence boundaries. It introduces jp-evalb, an alignment-based method that pre-aligns sentences and words to enable robust PARSEVAL measurement in end-to-end parsing pipelines, including cases with tokenization or boundary mismatches. The authors detail the algorithm, mismatch handling (word and sentence), and key assumptions, and demonstrate through case studies on Penn Treebank Section 23, evalb bug scenarios, and Korean end-to-end parsing that jp-evalb reproduces evalb results when desired while offering improved handling of mismatches. The work provides an open-source implementation, preserving compatibility with PARSEVAL while enabling broader, more realistic evaluation in contemporary parsing systems.

Abstract

We introduce an evaluation system designed to compute PARSEVAL measures, offering a viable alternative to \texttt{evalb} commonly used for constituency parsing evaluation. The widely used \texttt{evalb} script has traditionally been employed for evaluating the accuracy of constituency parsing results, albeit with the requirement for consistent tokenization and sentence boundaries. In contrast, our approach, named \texttt{jp-evalb}, is founded on an alignment method. This method aligns sentences and words when discrepancies arise. It aims to overcome several known issues associated with \texttt{evalb} by utilizing the `jointly preprocessed (JP)' alignment-based method. We introduce a more flexible and adaptive framework, ultimately contributing to a more accurate assessment of constituency parsing performance.

jp-evalb: Robust Alignment-based PARSEVAL Measures

TL;DR

This paper addresses limitations of the traditional PARSEVAL evaluation (evalb) which assumes exact tokenization and sentence boundaries. It introduces jp-evalb, an alignment-based method that pre-aligns sentences and words to enable robust PARSEVAL measurement in end-to-end parsing pipelines, including cases with tokenization or boundary mismatches. The authors detail the algorithm, mismatch handling (word and sentence), and key assumptions, and demonstrate through case studies on Penn Treebank Section 23, evalb bug scenarios, and Korean end-to-end parsing that jp-evalb reproduces evalb results when desired while offering improved handling of mismatches. The work provides an open-source implementation, preserving compatibility with PARSEVAL while enabling broader, more realistic evaluation in contemporary parsing systems.

Abstract

We introduce an evaluation system designed to compute PARSEVAL measures, offering a viable alternative to \texttt{evalb} commonly used for constituency parsing evaluation. The widely used \texttt{evalb} script has traditionally been employed for evaluating the accuracy of constituency parsing results, albeit with the requirement for consistent tokenization and sentence boundaries. In contrast, our approach, named \texttt{jp-evalb}, is founded on an alignment method. This method aligns sentences and words when discrepancies arise. It aims to overcome several known issues associated with \texttt{evalb} by utilizing the `jointly preprocessed (JP)' alignment-based method. We introduce a more flexible and adaptive framework, ultimately contributing to a more accurate assessment of constituency parsing performance.
Paper Structure (13 sections, 2 equations, 5 figures, 2 algorithms)

This paper contains 13 sections, 2 equations, 5 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Detailing the jp-evalb Algorithm
  3. Word and Sentence Mismatches
  4. Word mismatch
  5. Sentence mismatch
  6. Assumptions
  7. Usage of jp-evalb
  8. Case Studies
  9. Section 23 of the English Penn treebank
  10. Bug cases identified by evalb
  11. Korean end-to-end parsing evaluation
  12. Previous Work
  13. Conclusion

Figures (5)

  • Figure 1: Example of word and sentence mismatches
  • Figure 2: Difference between jp-evalb and evalb
  • Figure 3: Examples of evaluation results on Section 23 of the English Penn treebank
  • Figure 4: Evaluation results of bug cases by evalb
  • Figure 5: Evaluation results of the end-to-end Korean constituency parsing