Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

Eduard Poesina, Cornelia Caragea, Radu Tudor Ionescu

TL;DR

The first Romanian NLI corpus (RoNLI) is introduced comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels.

Abstract

Natural language inference (NLI), the task of recognizing the entailment relationship in sentence pairs, is an actively studied topic serving as a proxy for natural language understanding. Despite the relevance of the task in building conversational agents and improving text classification, machine translation and other NLP tasks, to the best of our knowledge, there is no publicly available NLI corpus for the Romanian language. To this end, we introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels. We conduct experiments with multiple machine learning methods based on distant learning, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. Furthermore, we improve on the best model by employing a new curriculum learning strategy based on data cartography. Our dataset and code to reproduce the baselines are available at https://github.com/Eduard6421/RONLI.

A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

TL;DR

The first Romanian NLI corpus (RoNLI) is introduced comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels.

Abstract

Natural language inference (NLI), the task of recognizing the entailment relationship in sentence pairs, is an actively studied topic serving as a proxy for natural language understanding. Despite the relevance of the task in building conversational agents and improving text classification, machine translation and other NLP tasks, to the best of our knowledge, there is no publicly available NLI corpus for the Romanian language. To this end, we introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels. We conduct experiments with multiple machine learning methods based on distant learning, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. Furthermore, we improve on the best model by employing a new curriculum learning strategy based on data cartography. Our dataset and code to reproduce the baselines are available at https://github.com/Eduard6421/RONLI.
Paper Structure (37 sections, 1 equation, 1 figure, 13 tables)

This paper contains 37 sections, 1 equation, 1 figure, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Corpus
  4. Data gathering and automatic labeling.
  5. Manual labeling.
  6. Data cartography.
  7. Models
  8. Classifiers based on word embeddings.
  9. RoGPT2.
  10. ChatGPT 3.5 Turbo.
  11. Multilingual BERT.
  12. Ro-BERT.
  13. Ro-BERT (spurious).
  14. Ro-BERT + cartography subsampling.
  15. Ro-BERT + Length-CL.
  16. ...and 22 more sections

Figures (1)

  • Figure 1: Data cartography visualization of the RoNLI dataset based on fine-tuning the Ro-BERT model Dumitrescu-EMNLP-2020. In the left-hand side plot, the $y$-axis corresponds to the level of confidence exhibited by the model during training, while the $x$-axis represents the variability of the confidence level. Adjacent to the primary plot, three histograms are displayed on the right-hand side, each representing a different metric: the confidence, the variability of confidence, and the correctness. The visualization offers a comprehensive overview of our dataset characteristics and the behavior of Ro-BERT during training. Best viewed in color.