Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Share What You Already Know: Cross-Language-Script Transfer and Alignment for Sentiment Detection in Code-Mixed Data

Niraj Pahari, Kazutaka Shimada

TL;DR

This work tackles sentiment analysis for code-mixed text written in different scripts by preserving script-specific pre-trained knowledge and enabling cross-language-script transfer. A three-component architecture employs script-specific encoders, cross-attention for cross-language sharing, and Earth Mover's Distance-based alignment plus regularization to align representations without losing script-specific semantic content. Experiments on Nepali-English and Hindi-English romanized datasets show the proposed method outperforms strong baselines, with ablations confirming the value of alignment and regularization, and SHAP analyses providing interpretability of cross-script knowledge sharing. The results suggest that cross-language-script transfer is a promising direction for code-mixed NLP and highlights the importance of data cleaning and robust transliteration for further gains.

Abstract

Code-switching entails mixing multiple languages. It is an increasingly occurring phenomenon in social media texts. Usually, code-mixed texts are written in a single script, even though the languages involved have different scripts. Pre-trained multilingual models primarily utilize the data in the native script of the language. In existing studies, the code-switched texts are utilized as they are. However, using the native script for each language can generate better representations of the text owing to the pre-trained knowledge. Therefore, a cross-language-script knowledge sharing architecture utilizing the cross attention and alignment of the representations of text in individual language scripts was proposed in this study. Experimental results on two different datasets containing Nepali-English and Hindi-English code-switched texts, demonstrate the effectiveness of the proposed method. The interpretation of the model using model explainability technique illustrates the sharing of language-specific knowledge between language-specific representations.

Share What You Already Know: Cross-Language-Script Transfer and Alignment for Sentiment Detection in Code-Mixed Data

TL;DR

This work tackles sentiment analysis for code-mixed text written in different scripts by preserving script-specific pre-trained knowledge and enabling cross-language-script transfer. A three-component architecture employs script-specific encoders, cross-attention for cross-language sharing, and Earth Mover's Distance-based alignment plus regularization to align representations without losing script-specific semantic content. Experiments on Nepali-English and Hindi-English romanized datasets show the proposed method outperforms strong baselines, with ablations confirming the value of alignment and regularization, and SHAP analyses providing interpretability of cross-script knowledge sharing. The results suggest that cross-language-script transfer is a promising direction for code-mixed NLP and highlights the importance of data cleaning and robust transliteration for further gains.

Abstract

Code-switching entails mixing multiple languages. It is an increasingly occurring phenomenon in social media texts. Usually, code-mixed texts are written in a single script, even though the languages involved have different scripts. Pre-trained multilingual models primarily utilize the data in the native script of the language. In existing studies, the code-switched texts are utilized as they are. However, using the native script for each language can generate better representations of the text owing to the pre-trained knowledge. Therefore, a cross-language-script knowledge sharing architecture utilizing the cross attention and alignment of the representations of text in individual language scripts was proposed in this study. Experimental results on two different datasets containing Nepali-English and Hindi-English code-switched texts, demonstrate the effectiveness of the proposed method. The interpretation of the model using model explainability technique illustrates the sharing of language-specific knowledge between language-specific representations.
Paper Structure (20 sections, 12 equations, 3 figures, 6 tables)

This paper contains 20 sections, 12 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Language-Script Specific Multi-Encoder Model
  4. BERT-Based Multilingual Model
  5. Cross-Language-Script Transfer of Representations
  6. Cross-Script Alignment
  7. Regularization
  8. Final Training Objective
  9. Model Interpretation
  10. Experiments
  11. Datasets
  12. Experimental Settings
  13. Hyperparameter Tuning
  14. Results and Discussions
  15. Results of Proposed Method
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Code-mixing example: Nepali-English pair and its English translation.
  • Figure 2: Proposed architecture of cross-language-script transfer multi-encoder model. The architecture comprises three modules: i. Cross-language-script transfer, as explained in Subsection \ref{['sub:cross-language-script-transfer']}, ii. Cross-script alignment, as explained in Subsection \ref{['sub:cross-script-alignment']}, and iii. Regularization, as explained in Subsection \ref{['sub:regularization']}.
  • Figure 3: Performance of the proposed model with different $i$-th layer representation for alignment and regularization.