Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language

Prathamesh Devadiga, Paras Chopra

TL;DR

This case study on Tulu systematically tackles various challenges posed by absence of training data for Tulu by combining explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play.

Abstract

Can large language models converse in languages virtually absent from their training data? We investigate this question through a case study on Tulu, a Dravidian language with over 2 million speakers but minimal digital presence. Rather than fine-tuning an LLM, we examine whether structured prompts alone can elicit basic conversational ability under controlled prompting. We systematically tackle various challenges posed by absence of training data for Tulu by combining explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play. Evaluated on a manually curated held-out set across three LLMs (Gemini 2.0 Flash, GPT-4o, Llama 3.1 70B) and validated by native speakers, our approach reduces vocabulary contamination from 80% to 5% while achieving 85% grammatical accuracy. Cross-model analysis reveals that negative constraints provide consistent improvements (12--18 percentage points), while grammar documentation effects vary by model architecture (8--22 points).

Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language

TL;DR

This case study on Tulu systematically tackles various challenges posed by absence of training data for Tulu by combining explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play.

Abstract

Can large language models converse in languages virtually absent from their training data? We investigate this question through a case study on Tulu, a Dravidian language with over 2 million speakers but minimal digital presence. Rather than fine-tuning an LLM, we examine whether structured prompts alone can elicit basic conversational ability under controlled prompting. We systematically tackle various challenges posed by absence of training data for Tulu by combining explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play. Evaluated on a manually curated held-out set across three LLMs (Gemini 2.0 Flash, GPT-4o, Llama 3.1 70B) and validated by native speakers, our approach reduces vocabulary contamination from 80% to 5% while achieving 85% grammatical accuracy. Cross-model analysis reveals that negative constraints provide consistent improvements (12--18 percentage points), while grammar documentation effects vary by model architecture (8--22 points).
Paper Structure (41 sections, 4 figures, 6 tables)

This paper contains 41 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-Resource NLP
  4. Constrained Generation
  5. Multilingual Prompting
  6. Script and Tokenization
  7. Methodology
  8. Romanization System
  9. Grammar Documentation
  10. Iterative Prompt Engineering
  11. Evaluation
  12. Results
  13. Main Results
  14. Ablation Studies
  15. Human Evaluation
  16. ...and 26 more sections

Figures (4)

  • Figure 1: Iterative improvement across four versions. Grammar accuracy increases from 18% to 85%, contamination decreases from 80% to 5%.
  • Figure 2: Five-layer prompt architecture with token counts. Negative constraints are positioned early (Layer 2) for maximum salience. Total: 2,800 tokens.
  • Figure 3: Component effectiveness heatmap across models. Negative constraints provide consistent improvements (12-18pp), while grammar documentation effects vary by architecture (8-22pp). Darker colors indicate larger improvements.
  • Figure 4: Five-layer prompt architecture. Components are ordered by salience priority.