Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages

TL;DR

The paper investigates word-level language identification in code-mixed Dravidian text using a prompt-based approach with GPT-3.5 Turbo, focusing on Kannada and Tamil. By evaluating zero-shot prompts at multiple temperatures, the study reveals Kannada achieves higher accuracy and reliability than Tamil, highlighting the persistent challenges posed by script variation and data scarcity in low-resource languages. The work contributes by applying prompt engineering to LI in Dravidian languages, detailing language-specific prompts and a structured evaluation across metrics such as Macro and Weighted F1, precision, recall, and accuracy. The findings underscore the potential of LLM-driven prompting for under-resourced language processing while pointing to avenues for prompt optimization and dataset expansion to close performance gaps in code-mixed contexts.

Abstract

Language Identification (LI) is crucial for various natural language processing tasks, serving as a foundational step in applications such as sentiment analysis, machine translation, and information retrieval. In multilingual societies like India, particularly among the youth engaging on social media, text often exhibits code-mixing, blending local languages with English at different linguistic levels. This phenomenon presents formidable challenges for LI systems, especially when languages intermingle within single words. Dravidian languages, prevalent in southern India, possess rich morphological structures yet suffer from under-representation in digital platforms, leading to the adoption of Roman or hybrid scripts for communication. This paper introduces a prompt based method for a shared task aimed at addressing word-level LI challenges in Dravidian languages. In this work, we leveraged GPT-3.5 Turbo to understand whether the large language models is able to correctly classify words into correct categories. Our findings show that the Kannada model consistently outperformed the Tamil model across most metrics, indicating a higher accuracy and reliability in identifying and categorizing Kannada language instances. In contrast, the Tamil model showed moderate performance, particularly needing improvement in precision and recall.

Figures (3)

• Figure 1: Dataset structure and word categories for the task
• Figure 2: An overview of GPT-3.5 Turbo for Kannada code-mixed language classification.
• Figure 3: An overview of GPT-3.5 Turbo for Tamil code-mixed language classification.