From Phonemes to Meaning: Evaluating Large Language Models on Tamil

TL;DR

This work introduces ILAKKANAM, a Tamil-linguistic benchmark of 820 questions drawn from Sri Lankan school examinations, annotated across five linguistic categories (Phonetics, Phonology, Morphology, Syntax, Semantics) plus a factual knowledge category. Using a standardized pipeline implemented via Abacus.AI, the authors evaluate both closed- and open-source large language models, revealing a consistent gap favoring closed models and a notable drop in performance as linguistic complexity increases. Gemini 2.5 emerges as the top performer overall, but linguistic-category tagging experiments show that the highest overall accuracy may reflect broad exposure rather than true linguistic grounding, underscoring a need for deeper, linguistically informed evaluation in Tamil. The dataset and analyses illuminate specific weak spots—especially in phonology and morphology—and provide a resource for guiding future improvements in Tamil language modeling and evaluation.

Abstract

Large Language Models (LLMs) have shown strong generalization across tasks in high-resource languages; however, their linguistic competence in low-resource and morphologically rich languages such as Tamil remains largely unexplored. Existing multilingual benchmarks often rely on translated English datasets, failing to capture the linguistic and cultural nuances of the target language. To address this gap, we introduce ILAKKANAM, the first Tamil-specific linguistic evaluation benchmark manually curated using 820 questions from Sri Lankan school-level Tamil subject examination papers. Each question is annotated by trained linguists under five linguistic categories and a factual knowledge category, spanning Grades 1--13 to ensure broad linguistic coverage. We evaluate both closed-source and open-source LLMs using a standardized evaluation framework. Our results show that Gemini 2.5 achieves the highest overall performance, while open-source models lag behind, highlighting the gap in linguistic grounding. Category- and grade-wise analyses reveal that all models perform well on lower-grade questions but show a clear decline as linguistic complexity increases. Further, no strong correlation is observed between a model's overall performance and its ability to identify linguistic categories, suggesting that performance may be driven by exposure rather than genuine understanding.

Figures (1)

• Figure 1: Performance comparison of LLMs. (a) Category-wise results showing linguistic variation. (b) Grade-wise results showing variation across difficulty levels.