Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NLPre: a revised approach towards language-centric benchmarking of Natural Language Preprocessing systems

Martyna Wiącek, Piotr Rybak, Łukasz Pszenny, Alina Wróblewska

TL;DR

This work investigates a novel method of reliable and fair evaluation and performance reporting of multiple NLPre tools, inspired by the GLUE benchmark, and enables comprehensive ongoing evaluation of multiple NLPre tools, while credibly tracking their performance.

Abstract

With the advancements of transformer-based architectures, we observe the rise of natural language preprocessing (NLPre) tools capable of solving preliminary NLP tasks (e.g. tokenisation, part-of-speech tagging, dependency parsing, or morphological analysis) without any external linguistic guidance. It is arduous to compare novel solutions to well-entrenched preprocessing toolkits, relying on rule-based morphological analysers or dictionaries. Aware of the shortcomings of existing NLPre evaluation approaches, we investigate a novel method of reliable and fair evaluation and performance reporting. Inspired by the GLUE benchmark, the proposed language-centric benchmarking system enables comprehensive ongoing evaluation of multiple NLPre tools, while credibly tracking their performance. The prototype application is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. Based on this benchmark, we conduct an extensive evaluation of a variety of Polish NLPre systems. To facilitate the construction of benchmarking environments for other languages, e.g. NLPre-GA for Irish or NLPre-ZH for Chinese, we ensure full customization of the publicly released source code of the benchmarking system. The links to all the resources (deployed platforms, source code, trained models, datasets etc.) can be found on the project website: https://sites.google.com/view/nlpre-benchmark.

NLPre: a revised approach towards language-centric benchmarking of Natural Language Preprocessing systems

TL;DR

This work investigates a novel method of reliable and fair evaluation and performance reporting of multiple NLPre tools, inspired by the GLUE benchmark, and enables comprehensive ongoing evaluation of multiple NLPre tools, while credibly tracking their performance.

Abstract

With the advancements of transformer-based architectures, we observe the rise of natural language preprocessing (NLPre) tools capable of solving preliminary NLP tasks (e.g. tokenisation, part-of-speech tagging, dependency parsing, or morphological analysis) without any external linguistic guidance. It is arduous to compare novel solutions to well-entrenched preprocessing toolkits, relying on rule-based morphological analysers or dictionaries. Aware of the shortcomings of existing NLPre evaluation approaches, we investigate a novel method of reliable and fair evaluation and performance reporting. Inspired by the GLUE benchmark, the proposed language-centric benchmarking system enables comprehensive ongoing evaluation of multiple NLPre tools, while credibly tracking their performance. The prototype application is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. Based on this benchmark, we conduct an extensive evaluation of a variety of Polish NLPre systems. To facilitate the construction of benchmarking environments for other languages, e.g. NLPre-GA for Irish or NLPre-ZH for Chinese, we ensure full customization of the publicly released source code of the benchmarking system. The links to all the resources (deployed platforms, source code, trained models, datasets etc.) can be found on the project website: https://sites.google.com/view/nlpre-benchmark.
Paper Structure (38 sections, 5 figures, 7 tables)

This paper contains 38 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction and related works
  2. NLPre benchmarking
  3. Research concept
  4. Online benchmarking system
  5. Configuration
  6. NLPre-PL benchmark
  7. Datasets
  8. NKJP1M
  9. PDB-UD
  10. Tasks
  11. Segmentation
  12. Tagging
  13. Lemmatisation
  14. Dependency parsing
  15. Evaluation
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Screenshot of the NLPre-PL leaderboard.
  • Figure 2: Pearson correlation coefficients between vectors of F1 scores on Tokens, Sentences, Words, UPOS, XPOS, Lemmas tasks averaged over datasets (excluding PDB-UD) and embeddings.
  • Figure 3: Pearson correlation coefficients between vectors of F1 scores on Tokens, Sentences, Words, UPOS, XPOS, Lemmas tasks averaged over datasets (excluding PDB-UD).
  • Figure 4: Dispersion of model performance measured by F1 on the Morfeusz tagset and Sentences, Words, UPOS, XPOS, and Lemmas tasks.
  • Figure 5: Dispersion of model performance measured by F1 on the UD tagset and Sentences, Words, UPOS, XPOS, and Lemmas tasks.