Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek

John Pavlopoulos, Juli Bakagianni, Kanella Pouli, Maria Gavriilidou

TL;DR

This work addresses NLP for a lesser-resourced language by focusing on Modern Greek, compiling a comprehensive, openly licensed Greek data collection, and benchmarking seven core tasks with both open (Llama-70b) and closed (GPT-4o mini) LLMs in 0-shot settings. It reveals task-dependent model strengths, with Llama excelling in NER and Summarization and GPT-4o mini leading in GEC, MT, Intent Classification, and POS tagging, while both are on par for toxicity detection. The study further reframes Authorship Attribution as a data-provenance probe for possible pre-training data leakage and introduces a first long Greek legal text clustering benchmark using STE representations that outperform TF-IDF baselines. Altogether, the results offer a practical roadmap for advancing NLP in Greek and other lesser-resourced languages, emphasizing dataset FAIRness, task innovation, and ethical data usage.

Abstract

Natural Language Processing (NLP) for lesser-resourced languages faces persistent challenges, including limited datasets, inherited biases from high-resource languages, and the need for domain-specific solutions. This study addresses these gaps for Modern Greek through three key contributions. First, we evaluate the performance of open-source (Llama-70b) and closed-source (GPT-4o mini) large language models (LLMs) on seven core NLP tasks with dataset availability, revealing task-specific strengths, weaknesses, and parity in their performance. Second, we expand the scope of Greek NLP by reframing Authorship Attribution as a tool to assess potential data usage by LLMs in pre-training, with high 0-shot accuracy suggesting ethical implications for data provenance. Third, we showcase a legal NLP case study, where a Summarize, Translate, and Embed (STE) methodology outperforms the traditional TF-IDF approach for clustering \emph{long} legal texts. Together, these contributions provide a roadmap to advance NLP in lesser-resourced languages, bridging gaps in model evaluation, task innovation, and real-world impact.

Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek

TL;DR

This work addresses NLP for a lesser-resourced language by focusing on Modern Greek, compiling a comprehensive, openly licensed Greek data collection, and benchmarking seven core tasks with both open (Llama-70b) and closed (GPT-4o mini) LLMs in 0-shot settings. It reveals task-dependent model strengths, with Llama excelling in NER and Summarization and GPT-4o mini leading in GEC, MT, Intent Classification, and POS tagging, while both are on par for toxicity detection. The study further reframes Authorship Attribution as a data-provenance probe for possible pre-training data leakage and introduces a first long Greek legal text clustering benchmark using STE representations that outperform TF-IDF baselines. Altogether, the results offer a practical roadmap for advancing NLP in Greek and other lesser-resourced languages, emphasizing dataset FAIRness, task innovation, and ethical data usage.

Abstract

Natural Language Processing (NLP) for lesser-resourced languages faces persistent challenges, including limited datasets, inherited biases from high-resource languages, and the need for domain-specific solutions. This study addresses these gaps for Modern Greek through three key contributions. First, we evaluate the performance of open-source (Llama-70b) and closed-source (GPT-4o mini) large language models (LLMs) on seven core NLP tasks with dataset availability, revealing task-specific strengths, weaknesses, and parity in their performance. Second, we expand the scope of Greek NLP by reframing Authorship Attribution as a tool to assess potential data usage by LLMs in pre-training, with high 0-shot accuracy suggesting ethical implications for data provenance. Third, we showcase a legal NLP case study, where a Summarize, Translate, and Embed (STE) methodology outperforms the traditional TF-IDF approach for clustering \emph{long} legal texts. Together, these contributions provide a roadmap to advance NLP in lesser-resourced languages, bridging gaps in model evaluation, task innovation, and real-world impact.
Paper Structure (44 sections, 4 figures, 18 tables)

This paper contains 44 sections, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. The data
  3. Availability, Popularity and Licensing
  4. Distilling FAIR data
  5. Datasets with no supervision signal
  6. Dataset statistics
  7. Linguistic distance
  8. Datasets with supervision signal
  9. The Greek NLP Benchmark
  10. Toxicity Detection
  11. The dataset
  12. The results
  13. Grammatical Error Correction
  14. The dataset
  15. The results
  16. ...and 29 more sections

Figures (4)

  • Figure 1: Availability and accessibility of extracted datasets, one tuple per bar. Availability is classified to open (yes), constrained (lmt), hindered by an error (err), or not available via a URL (no). Accessibility reflects the outcome we observed when accessing the resource.
  • Figure 2: Whisker and box plot with the number of citations of studies per availability type.
  • Figure 3: Citation counts per year of publication of studies developing datasets, based on the availability classified as yes, no, limited (lmt), or erroneous (err).
  • Figure 4: Heatmap of BPC measured on a sample per dataset (average across the sample; shown horizontally) per trained LM (vertically). Distance in warm colours.