Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding the Diversity: A Review of the Indic AI Research Landscape

Sankalp KJ, Vinija Jain, Sreyoshi Bhaduri, Tamoghna Roy, Aman Chadha

TL;DR

This work aims to serve as a valuable resource for researchers and practitioners working in the field of NLP, particularly those focused on Indic languages, and contributes to the development of more accurate and efficient LLM applications for these languages.

Abstract

This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and Bhutan, among others. These languages have a rich cultural and linguistic heritage and are spoken by over 1.5 billion people worldwide. With the tremendous market potential and growing demand for natural language processing (NLP) based applications in diverse languages, generative applications for Indic languages pose unique challenges and opportunities for research. Our paper deep dives into the recent advancements in Indic generative modeling, contributing with a taxonomy of research directions, tabulating 84 recent publications. Research directions surveyed in this paper include LLM development, fine-tuning existing LLMs, development of corpora, benchmarking and evaluation, as well as publications around specific techniques, tools, and applications. We found that researchers across the publications emphasize the challenges associated with limited data availability, lack of standardization, and the peculiar linguistic complexities of Indic languages. This work aims to serve as a valuable resource for researchers and practitioners working in the field of NLP, particularly those focused on Indic languages, and contributes to the development of more accurate and efficient LLM applications for these languages.

Decoding the Diversity: A Review of the Indic AI Research Landscape

TL;DR

This work aims to serve as a valuable resource for researchers and practitioners working in the field of NLP, particularly those focused on Indic languages, and contributes to the development of more accurate and efficient LLM applications for these languages.

Abstract

This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and Bhutan, among others. These languages have a rich cultural and linguistic heritage and are spoken by over 1.5 billion people worldwide. With the tremendous market potential and growing demand for natural language processing (NLP) based applications in diverse languages, generative applications for Indic languages pose unique challenges and opportunities for research. Our paper deep dives into the recent advancements in Indic generative modeling, contributing with a taxonomy of research directions, tabulating 84 recent publications. Research directions surveyed in this paper include LLM development, fine-tuning existing LLMs, development of corpora, benchmarking and evaluation, as well as publications around specific techniques, tools, and applications. We found that researchers across the publications emphasize the challenges associated with limited data availability, lack of standardization, and the peculiar linguistic complexities of Indic languages. This work aims to serve as a valuable resource for researchers and practitioners working in the field of NLP, particularly those focused on Indic languages, and contributes to the development of more accurate and efficient LLM applications for these languages.
Paper Structure (25 sections, 1 figure)

This paper contains 25 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Methodology
  3. Search Strategy
  4. Inclusion and Exclusion Criteria
  5. Data Extraction and Synthesis
  6. Analysis and Taxonomy Development
  7. Results
  8. LLMs
  9. Pre-trained LLMs
  10. Fine-tuned LLMs
  11. Corpora
  12. Multi-lingual corpora
  13. Language specific corpora
  14. Benchmarks and Evaluation
  15. Multi-lingual performance evaluation
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Taxonomy of Indic AI Research: This taxonomy provides a structured overview of Indic AI research, detailing the creation and fine-tuning of large language models and their specific applications. It categorizes the research into several branches, including pre-trained and fine-tuned LLMs, multilingual and language-specific corpora, evaluation techniques, and various tools and applications. Each category lists relevant models, datasets, and methodologies, illustrating the comprehensive landscape of Indic AI advancements.