Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Literary and Colloquial Tamil Dialect Identification

M. Nanmalar, P. Vijayalakshmi, T. Nagarajan

TL;DR

The paper addresses the problem of distinguishing literary versus colloquial Tamil as a prerequisite for LT↔CT conversion in language technologies. It presents six dialect-identification approaches spanning implicit (GMM, CNN) and explicit (PPR, P-LVCSR) categories, plus two unified explicit variants (UPR-1 and UPR-2), evaluated on a purpose-built LT/CT corpus with text and speech data. Results show strong performance across methods, with CNN achieving 93.97% and P-LVCSR 94.21%, and the proposed UPR-2 plus P-LVCSR reaching the top accuracy of 95.61%, after addressing practical issues in parallel word handling and alignment. The work demonstrates that LT and CT can be effectively distinguished using both acoustic and phonetic-lexical cues, offering practical pathways for LT–CT conversion and Tamil language technology deployment, especially in environments with limited annotated data.

Abstract

Culture and language evolve together. The old literary form of Tamil is used commonly for writing and the contemporary colloquial Tamil is used for speaking. Human-computer interaction applications require Colloquial Tamil (CT) to make it more accessible and easy for the everyday user and, it requires Literary Tamil (LT) when information is needed in a formal written format. Continuing the use of LT alongside CT in computer aided language learning applications will both preserve LT, and provide ease of use via CT, at the same time. Hence there is a need for the conversion between LT and CT dialects, which demands as a first step, dialect identification. Dialect Identification (DID) of LT and CT is an unexplored area of research. In the current work, keeping the nuances of both these dialects in mind, five methods are explored which include two implicit methods - Gaussian Mixture Model (GMM) and Convolutional Neural Network (CNN); two explicit methods - Parallel Phone Recognition (PPR) and Parallel Large Vocabulary Continuous Speech Recognition (P-LVCSR); two versions of the proposed explicit Unified Phone Recognition method (UPR-1 and UPR-2). These methods vary based on: the need for annotated data, the size of the unit, the way in which modelling is carried out, and the way in which the final decision is made. Even though the average duration of the test utterances is less - 4.9s for LT and 2.5s for CT - the systems performed well, offering the following identification accuracies: 87.72% (GMM), 93.97% (CNN), 89.24% (PPR), 94.21% (P-LVCSR), 88.57% (UPR-1), 93.53% (UPR-1 with P-LVCSR), 94.55% (UPR-2), and 95.61% (UPR-2 with P-LVCSR).

Literary and Colloquial Tamil Dialect Identification

TL;DR

The paper addresses the problem of distinguishing literary versus colloquial Tamil as a prerequisite for LT↔CT conversion in language technologies. It presents six dialect-identification approaches spanning implicit (GMM, CNN) and explicit (PPR, P-LVCSR) categories, plus two unified explicit variants (UPR-1 and UPR-2), evaluated on a purpose-built LT/CT corpus with text and speech data. Results show strong performance across methods, with CNN achieving 93.97% and P-LVCSR 94.21%, and the proposed UPR-2 plus P-LVCSR reaching the top accuracy of 95.61%, after addressing practical issues in parallel word handling and alignment. The work demonstrates that LT and CT can be effectively distinguished using both acoustic and phonetic-lexical cues, offering practical pathways for LT–CT conversion and Tamil language technology deployment, especially in environments with limited annotated data.

Abstract

Culture and language evolve together. The old literary form of Tamil is used commonly for writing and the contemporary colloquial Tamil is used for speaking. Human-computer interaction applications require Colloquial Tamil (CT) to make it more accessible and easy for the everyday user and, it requires Literary Tamil (LT) when information is needed in a formal written format. Continuing the use of LT alongside CT in computer aided language learning applications will both preserve LT, and provide ease of use via CT, at the same time. Hence there is a need for the conversion between LT and CT dialects, which demands as a first step, dialect identification. Dialect Identification (DID) of LT and CT is an unexplored area of research. In the current work, keeping the nuances of both these dialects in mind, five methods are explored which include two implicit methods - Gaussian Mixture Model (GMM) and Convolutional Neural Network (CNN); two explicit methods - Parallel Phone Recognition (PPR) and Parallel Large Vocabulary Continuous Speech Recognition (P-LVCSR); two versions of the proposed explicit Unified Phone Recognition method (UPR-1 and UPR-2). These methods vary based on: the need for annotated data, the size of the unit, the way in which modelling is carried out, and the way in which the final decision is made. Even though the average duration of the test utterances is less - 4.9s for LT and 2.5s for CT - the systems performed well, offering the following identification accuracies: 87.72% (GMM), 93.97% (CNN), 89.24% (PPR), 94.21% (P-LVCSR), 88.57% (UPR-1), 93.53% (UPR-1 with P-LVCSR), 94.55% (UPR-2), and 95.61% (UPR-2 with P-LVCSR).
Paper Structure (24 sections, 10 equations, 6 figures, 6 tables)

This paper contains 24 sections, 10 equations, 6 figures, 6 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Nuances of Literary and Colloquial Tamil
  4. Corpus
  5. Text Corpus
  6. Speech Corpus
  7. Implicit Systems
  8. Dialect Identification using GMM
  9. Dialect Identification using CNN
  10. Explicit Systems
  11. Dialect Identification using PPR
  12. Version 1 - Conventional PPR
  13. Version 2 - Unique Phone Inclusion
  14. Version 3 - Language Model Exclusion
  15. Dialect Identification using P-LVCSR
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Differences between literary and colloquial Tamil
  • Figure 2: Histogram of LT and CT Utterance Sizes
  • Figure 3: Block Diagram of the proposed 1D-CNN architecture for LT/CT DID
  • Figure 4: Block Diagram of a PPR System
  • Figure 5: Block Diagram of a P-LVCSR System
  • ...and 1 more figures