Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers

Roy Xie, Orevaoghene Ahia, Yulia Tsvetkov, Antonios Anastasopoulos

TL;DR

The paper tackles the challenge of identifying dialect-specific lexical features without relying on expert linguists by leveraging interpretable dialect classifiers. It combines post-hoc (Leave-One-Out) and intrinsic (SelfExplain) explanation methods to extract local explanations and maps them to global lexical features through a corpus-level TF-IDF workflow. Experiments on Mandarin CN-TW, Italian dialects (including Sicilian), and Low Saxon demonstrate high explanation sufficiency, favorable plausibility, and strong lexical-feature extraction performance, with automatic and human validations supporting the approach. This work provides a framework for explainable dialectology, enabling researchers to uncover language-unique vocabulary and offering practical insights for NLP applications involving dialectal variation.

Abstract

Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis. This is largely due to the complexity and nuance involved in studying various dialects. We present a novel approach to extract distinguishing lexical features of dialects by utilizing interpretable dialect classifiers, even in the absence of human experts. We explore both post-hoc and intrinsic approaches to interpretability, conduct experiments on Mandarin, Italian, and Low Saxon, and experimentally demonstrate that our method successfully identifies key language-specific lexical features that contribute to dialectal variations.

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers

TL;DR

The paper tackles the challenge of identifying dialect-specific lexical features without relying on expert linguists by leveraging interpretable dialect classifiers. It combines post-hoc (Leave-One-Out) and intrinsic (SelfExplain) explanation methods to extract local explanations and maps them to global lexical features through a corpus-level TF-IDF workflow. Experiments on Mandarin CN-TW, Italian dialects (including Sicilian), and Low Saxon demonstrate high explanation sufficiency, favorable plausibility, and strong lexical-feature extraction performance, with automatic and human validations supporting the approach. This work provides a framework for explainable dialectology, enabling researchers to uncover language-unique vocabulary and offering practical insights for NLP applications involving dialectal variation.

Abstract

Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis. This is largely due to the complexity and nuance involved in studying various dialects. We present a novel approach to extract distinguishing lexical features of dialects by utilizing interpretable dialect classifiers, even in the absence of human experts. We explore both post-hoc and intrinsic approaches to interpretability, conduct experiments on Mandarin, Italian, and Low Saxon, and experimentally demonstrate that our method successfully identifies key language-specific lexical features that contribute to dialectal variations.
Paper Structure (40 sections, 9 equations, 10 figures, 12 tables)

This paper contains 40 sections, 9 equations, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Method
  4. Interpretable Dialect Classifier
  5. Explanation Methods
  6. Post-hoc Approach
  7. Intrinsic Approach
  8. Mapping Explanations to Lexical Features
  9. Languages and Datasets
  10. Languages
  11. Datasets
  12. Experiments
  13. Models
  14. TF-IDF Baseline
  15. Explanation Evaluation
  16. ...and 25 more sections

Figures (10)

  • Figure 1: (1) given an input text; (2) the interpretable dialect classifier return labels (SCN and IT) and explanations; (3) the extractor takes the explanations and (4) outputs meaningful features to the languages.
  • Figure 2: Adequate justification percentage for LOO and SelfExplain. Humans found LOO produces more justifiable explanations across all four dialects.
  • Figure 3: A general overview of the geographical areas in Italy for the 11 languages and dialects. While the map's vague due to the complexity of the situation, it provides a rough idea of where in Italy to locate the varieties. The map is sourced from aepli-etal-2022-findings.
  • Figure 4: Rough regions where the 16 considered Low Saxon languages and dialects are spoken. This map is taken from siewert-etal-2020-lsdc.
  • Figure 5: SelfExplain CN Class feature counts in explanation and input text.
  • ...and 5 more figures