Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Essay Scoring and Feedback Generation in Basque Language Learning

Ekhi Azurmendi, Xabier Arregi, Oier Lopez de Lacalle

TL;DR

This work presents the first publicly available Basque C1 Automatic Essay Scoring (AES) dataset with detailed, criterion-specific feedback and error-examples. It evaluates encoder-based models (RoBERTa-EusCrawl) and Latxa models (8B/70B), showing that supervised fine-tuning (SFT) of Latxa markedly improves Correctness scoring and yields highly consistent, criterion-aligned feedback that surpasses several closed-source baselines. A novel evaluation framework combines automatic consistency metrics for feedback with expert manual validation of learner errors, demonstrating broader error-type coverage for the SFT Latxa model. The dataset and findings establish a transparent, reproducible benchmark for NLP in low-resource languages like Basque and point to broader educational benefits in CALL contexts.

Abstract

This paper introduces the first publicly available dataset for Automatic Essay Scoring (AES) and feedback generation in Basque, targeting the CEFR C1 proficiency level. The dataset comprises 3,200 essays from HABE, each annotated by expert evaluators with criterion specific scores covering correctness, richness, coherence, cohesion, and task alignment enriched with detailed feedback and error examples. We fine-tune open-source models, including RoBERTa-EusCrawl and Latxa 8B/70B, for both scoring and explanation generation. Our experiments show that encoder models remain highly reliable for AES, while supervised fine-tuning (SFT) of Latxa significantly enhances performance, surpassing state-of-the-art (SoTA) closed-source systems such as GPT-5 and Claude Sonnet 4.5 in scoring consistency and feedback quality. We also propose a novel evaluation methodology for assessing feedback generation, combining automatic consistency metrics with expert-based validation of extracted learner errors. Results demonstrate that the fine-tuned Latxa model produces criterion-aligned, pedagogically meaningful feedback and identifies a wider range of error types than proprietary models. This resource and benchmark establish a foundation for transparent, reproducible, and educationally grounded NLP research in low-resource languages such as Basque.

Automatic Essay Scoring and Feedback Generation in Basque Language Learning

TL;DR

This work presents the first publicly available Basque C1 Automatic Essay Scoring (AES) dataset with detailed, criterion-specific feedback and error-examples. It evaluates encoder-based models (RoBERTa-EusCrawl) and Latxa models (8B/70B), showing that supervised fine-tuning (SFT) of Latxa markedly improves Correctness scoring and yields highly consistent, criterion-aligned feedback that surpasses several closed-source baselines. A novel evaluation framework combines automatic consistency metrics for feedback with expert manual validation of learner errors, demonstrating broader error-type coverage for the SFT Latxa model. The dataset and findings establish a transparent, reproducible benchmark for NLP in low-resource languages like Basque and point to broader educational benefits in CALL contexts.

Abstract

This paper introduces the first publicly available dataset for Automatic Essay Scoring (AES) and feedback generation in Basque, targeting the CEFR C1 proficiency level. The dataset comprises 3,200 essays from HABE, each annotated by expert evaluators with criterion specific scores covering correctness, richness, coherence, cohesion, and task alignment enriched with detailed feedback and error examples. We fine-tune open-source models, including RoBERTa-EusCrawl and Latxa 8B/70B, for both scoring and explanation generation. Our experiments show that encoder models remain highly reliable for AES, while supervised fine-tuning (SFT) of Latxa significantly enhances performance, surpassing state-of-the-art (SoTA) closed-source systems such as GPT-5 and Claude Sonnet 4.5 in scoring consistency and feedback quality. We also propose a novel evaluation methodology for assessing feedback generation, combining automatic consistency metrics with expert-based validation of extracted learner errors. Results demonstrate that the fine-tuned Latxa model produces criterion-aligned, pedagogically meaningful feedback and identifies a wider range of error types than proprietary models. This resource and benchmark establish a foundation for transparent, reproducible, and educationally grounded NLP research in low-resource languages such as Basque.

Paper Structure

This paper contains 29 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. 1) Encoder-based classifiers remain competitive for AES.
  3. 2) Supervised fine-tuning (SFT) on Latxa improves correctness scoring.
  4. 3) Feedback and error-example generation enhance correctness scoring.
  5. 4) SFT Latxa demonstrates strong consistency between feedback and scores.
  6. 5) Training on the dataset improves coverage of error-example types
  7. Related work
  8. Automatic Essay Scoring
  9. Automatic Review Generation
  10. Basque Approaches in Education
  11. The Basque C1 Dataset
  12. Feedback and error-example annotation
  13. General overview
  14. Experimental setting
  15. AES setting
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Excerpt from the Basque C1 dataset. Each essay includes multiple annotations, where each criterion (task alignment, coherence, lexical richness, and correctness) is aligned with natural-language feedback (criterion feedback), a score from A to D (criterion score), and learner error (error-example). Finally, an overall score of the essay is provided.
  • Figure 2: Basic statistics of the Basque C1 dataset.
  • Figure 3: Category percentage and accuracy of models