Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer

Haeji Jung, Changdae Oh, Jooeon Kang, Jimin Sohn, Kyungwoo Song, Jinkyu Kim, David R. Mortensen

TL;DR

The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced.

Abstract

Approaches to improving multilingual language understanding often struggle with significant performance gaps between high-resource and low-resource languages. While there are efforts to align the languages in a single latent space to mitigate such gaps, how different input-level representations influence such gaps has not been investigated, particularly with phonemic inputs. We hypothesize that the performance gaps are affected by representation discrepancies between these languages, and revisit the use of phonemic representations as a means to mitigate these discrepancies. To demonstrate the effectiveness of phonemic representations, we present experiments on three representative cross-lingual tasks on 12 languages in total. The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced. We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representations, and it is further justified by a theoretical analysis of the cross-lingual performance gap.

Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer

TL;DR

The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced.

Abstract

Approaches to improving multilingual language understanding often struggle with significant performance gaps between high-resource and low-resource languages. While there are efforts to align the languages in a single latent space to mitigate such gaps, how different input-level representations influence such gaps has not been investigated, particularly with phonemic inputs. We hypothesize that the performance gaps are affected by representation discrepancies between these languages, and revisit the use of phonemic representations as a means to mitigate these discrepancies. To demonstrate the effectiveness of phonemic representations, we present experiments on three representative cross-lingual tasks on 12 languages in total. The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced. We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representations, and it is further justified by a theoretical analysis of the cross-lingual performance gap.
Paper Structure (30 sections, 3 theorems, 8 equations, 3 figures, 5 tables)

This paper contains 30 sections, 3 theorems, 8 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cross-lingual Transfer with Multilingual Language Model
  4. Cross-lingual Representation Gap
  5. Phonemic Representations for Multilingual Language Modeling
  6. Experimental Setup
  7. Models
  8. Downstream Tasks
  9. Token-level Classification.
  10. Sentence-level Classification.
  11. Performance Gap
  12. Linguistic Gap
  13. Implementation Details
  14. Data Preparation
  15. Languages.
  16. ...and 15 more sections

Key Result

Theorem 4.1

Let $h:\mathcal{X} \rightarrow \{0,1\}$ be a function in a hypothesis class $\mathcal{H}$ with a pseudo dimension $\mathcal{P} dim(\mathcal{H})=d$. If $\hat{\mathcal{D}}_{A}$ and $\hat{\mathcal{D}}_{B}$ are the empirical distribution constructed by $n$-size i.i.d. samples, drawn from $\mathcal{D}_{A

Figures (3)

  • Figure 1: Example of orthographic and phonemic input representations of a sentence (English and Korean).
  • Figure 2: Linguistic gaps across languages in each model. (Center) Upper and lower triangular elements of the heatmap indicate pairwise linguistic gaps derived with character-based model and phoneme-based model, respectively. Darker color indicates larger CKA score, which means smaller discrepancy. Lower triangular elements show relatively darker colors, implying smaller discrepancies across languages of phoneme-based model. (Left, right) T-SNE plots for each model are shown with only five languages, for better visibility.
  • Figure 3: Qualitative analysis of performance gap (difference of accruacy) on XNLI task. (Left) the absolute difference between performance across two languages, (center) centered kernel alignment (CKA) scores to measure cross-lingual embedding similarity, and (right) Sinkhorn distance on the output probability space. Phonemic representation shows relatively small performance gaps w.r.t. eng$\leftrightarrow$swa and eng$\leftrightarrow$urd, and these gaps are correlated with similarity and discrepancy on the embedding space (CKA) and logit space (Sinkhorn distance).

Theorems & Definitions (5)

  • Theorem 4.1
  • Definition C.1: $\mathcal{H}$-divergence; bendavid2006
  • Theorem C.2
  • proof : proof of Theorem B.2.
  • Lemma C.3