Sample-Efficient Language Model for Hinglish Conversational AI

TL;DR

This work addresses the challenge of building a conversational AI for Hinglish, a Hindi-English code-mixed language plagued by data scarcity and variable spelling. It evaluates multiple cross-lingual, pre-trained models and leverages synthetic Hinglish data generated via Gemini-2.0-Flash, combined with parameter-efficient fine-tuning (LoRA and QLoRA) to achieve sample-efficient Hinglish dialogue generation. The study finds that small- to mid-sized models (e.g., Qwen2.5-3B/7B) fine-tuned with LoRA can deliver competitive Hinglish performance in fluency, coherence, and user alignment, often approaching large-model benchmarks while reducing compute needs. This approach demonstrates a practical pathway to culturally aware, lightweight Hinglish conversational AI suitable for deployment in resource-constrained settings, with avenues for future multimodal extensions and evaluation refinements.

Abstract

This paper presents our process for developing a sample-efficient language model for a conversational Hinglish chatbot. Hinglish, a code-mixed language that combines Hindi and English, presents a unique computational challenge due to inconsistent spelling, lack of standardization, and limited quality of conversational data. This work evaluates multiple pre-trained cross-lingual language models, including Gemma3-4B and Qwen2.5-7B, and employs fine-tuning techniques to improve performance on Hinglish conversational tasks. The proposed approach integrates synthetically generated dialogues with insights from existing Hinglish datasets to address data scarcity. Experimental results demonstrate that models with fewer parameters, when appropriately fine-tuned on high-quality code-mixed data, can achieve competitive performance for Hinglish conversation generation while maintaining computational efficiency.

Figures (2)

• Figure 1: Surveyors' Model Preference
• Figure 2: Hinglish Chatbot Example Conversation & UI