Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OYXOY: A Modern NLP Test Suite for Modern Greek

Konstantinos Kogkalidis, Stergios Chatzikyriakidis, Eirini Chrysovalantou Giannikouri, Vassiliki Katsouli, Christina Klironomou, Christina Koula, Dimitris Papadakis, Thelka Pasparaki, Erofili Psaltaki, Efthymia Sakellariou, Hara Soupiona

TL;DR

OYXOY introduces a modern Greek NLP evaluation suite with four tasks that foreground lexical semantics, inference, and figurative language. It uses a novel all-inference-label NLI dataset and repurposes the Dictionary of Standard Modern Greek via ChatGPT to support WiC, sense selection, and metaphor detection tasks, accompanied by GreekBERT baselines. The results reveal substantial challenges in Greek NLP and the need for targeted progress and resource expansion. The work emphasizes open-source, scalable data generation for under-resourced languages and invites community-driven development and extension.

Abstract

This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just \textit{one}, but rather \textit{all} possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we transform the Dictionary of Standard Modern Greek into a structured format, from which we derive the other three tasks through simple projections. Alongside each task, we conduct experiments using currently available state of the art machinery. Our experimental baselines affirm the challenging nature of our tasks and highlight the need for expedited progress in order for the Greek NLP ecosystem to keep pace with contemporary mainstream research.

OYXOY: A Modern NLP Test Suite for Modern Greek

TL;DR

OYXOY introduces a modern Greek NLP evaluation suite with four tasks that foreground lexical semantics, inference, and figurative language. It uses a novel all-inference-label NLI dataset and repurposes the Dictionary of Standard Modern Greek via ChatGPT to support WiC, sense selection, and metaphor detection tasks, accompanied by GreekBERT baselines. The results reveal substantial challenges in Greek NLP and the need for targeted progress and resource expansion. The work emphasizes open-source, scalable data generation for under-resourced languages and invites community-driven development and extension.

Abstract

This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just \textit{one}, but rather \textit{all} possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we transform the Dictionary of Standard Modern Greek into a structured format, from which we derive the other three tasks through simple projections. Alongside each task, we conduct experiments using currently available state of the art machinery. Our experimental baselines affirm the challenging nature of our tasks and highlight the need for expedited progress in order for the Greek NLP ecosystem to keep pace with contemporary mainstream research.
Paper Structure (16 sections, 1 figure, 4 tables)

This paper contains 16 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. OYXOY
  3. Natural Language Inference
  4. Repurposing the Lexicon
  5. The Role of ChatGPT
  6. Words-in-Context
  7. Sense Selection
  8. Metaphor Detection
  9. Experimental Baselines
  10. Natural Language Inference
  11. Sense Disambiguation
  12. Words-in-Context
  13. Sense Selection
  14. Metaphor Detection
  15. Related Work
  16. ...and 1 more sections

Figures (1)

  • Figure 1: NLI examples 761 and 879, showcasing multiple inferences. In the first example, φιλώ [/filó/] (to kiss) can be a unidirectional or a reciprocal action (i.e., to give a kiss to vs. to exchange kisses with). In the second example, pro-drop allows for two possible readings, where either Giorgos or Maria can be the subject of the embedded clause.