Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finetuning LLMs for EvaCun 2025 token prediction shared task

Josef Jon, Ondřej Bojar

TL;DR

This work tackles token prediction for cuneiform-origin languages (Akkadian and Sumerian) by fine-tuning autoregressive LLMs on the EvaCun 2025 task data using three prompt-based formulations. The authors implement a purely engineering baseline with three models (Aya Expanse 8B, Command-R 34B, Mistral Small 24B) and compare prompts All, One by One, and Restore, evaluating on a held-out dev set. They report that the best single-model performance reaches around 22.1% accuracy, while ensembling via majority voting raises accuracy to about 26.9%, highlighting both the feasibility and limitations of LLM-based token restoration for these ancient languages. The study provides a solid baseline for future, more language-informed approaches and suggests directions toward direct translation of cuneiform or visual-reading LLMs, given dataset biases and repetitiveness. Overall, it demonstrates that prompt-based fine-tuning of autoregressive LLMs can yield meaningful token restoration performance and sets a benchmark for future improvements in cuneiform NLP.

Abstract

In this paper, we present our submission for the token prediction task of EvaCun 2025. Our sys-tems are based on LLMs (Command-R, Mistral, and Aya Expanse) fine-tuned on the task data provided by the organizers. As we only pos-sess a very superficial knowledge of the subject field and the languages of the task, we simply used the training data without any task-specific adjustments, preprocessing, or filtering. We compare 3 different approaches (based on 3 different prompts) of obtaining the predictions, and we evaluate them on a held-out part of the data.

Finetuning LLMs for EvaCun 2025 token prediction shared task

TL;DR

This work tackles token prediction for cuneiform-origin languages (Akkadian and Sumerian) by fine-tuning autoregressive LLMs on the EvaCun 2025 task data using three prompt-based formulations. The authors implement a purely engineering baseline with three models (Aya Expanse 8B, Command-R 34B, Mistral Small 24B) and compare prompts All, One by One, and Restore, evaluating on a held-out dev set. They report that the best single-model performance reaches around 22.1% accuracy, while ensembling via majority voting raises accuracy to about 26.9%, highlighting both the feasibility and limitations of LLM-based token restoration for these ancient languages. The study provides a solid baseline for future, more language-informed approaches and suggests directions toward direct translation of cuneiform or visual-reading LLMs, given dataset biases and repetitiveness. Overall, it demonstrates that prompt-based fine-tuning of autoregressive LLMs can yield meaningful token restoration performance and sets a benchmark for future improvements in cuneiform NLP.

Abstract

In this paper, we present our submission for the token prediction task of EvaCun 2025. Our sys-tems are based on LLMs (Command-R, Mistral, and Aya Expanse) fine-tuned on the task data provided by the organizers. As we only pos-sess a very superficial knowledge of the subject field and the languages of the task, we simply used the training data without any task-specific adjustments, preprocessing, or filtering. We compare 3 different approaches (based on 3 different prompts) of obtaining the predictions, and we evaluate them on a held-out part of the data.
Paper Structure (10 sections, 1 figure, 3 tables)

This paper contains 10 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Data preprocessing and prompts
  5. Experiments
  6. Data
  7. LLM finetuning
  8. Results
  9. Conclusion
  10. Acknowledgments

Figures (1)

  • Figure 1: Relative frequencies of the top-20 generated words for masked positions by best checkpoint for each prompt (All, One by one, Restore on top left, top right and bottom left, respectively) and of the reference masked words (bottom right).