Saving the legacy of Hero Ibash: Evaluating Four Language Models for Aminoacian
Yunze Xiao, Yiyang Pan
TL;DR
This work tackles the challenge of processing a low-resource language, Aminoas, with an emphasis on its distinctive OVS syntax and tonal system. It evaluates four leading LLMs (Llama2, ChatGPT, Mistral, Ernie-bot) on Aminoas through machine translation to Chinese, question answering, and entailment tasks, using a curated Aminoas–Chinese dataset and standard metrics. The main finding is that none of the models can reliably translate Aminoas, revealing a substantial gap in current NLP capabilities for underrepresented languages and underscoring the need for targeted data collection and novel modeling approaches. The study highlights the importance of inclusive language technologies and lays groundwork for future research in data augmentation, cross-lingual transfer, and interdisciplinary collaboration to preserve linguistic diversity and improve digital accessibility for Aminoas speakers.
Abstract
This study assesses four cutting-edge language models in the underexplored Aminoacian language. Through evaluation, it scrutinizes their adaptability, effectiveness, and limitations in text generation, semantic coherence, and contextual understanding. Uncovering insights into these models' performance in a low-resourced language, this research pioneers pathways to bridge linguistic gaps. By offering benchmarks and understanding challenges, it lays groundwork for future advancements in natural language processing, aiming to elevate the applicability of language models in similar linguistic landscapes, marking a significant step toward inclusivity and progress in language technology.