Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer

TL;DR

This work tackles unsupervised cross-lingual transfer by addressing limitations of single-kind linguistic cues. It introduces Lexicon-Syntax Enhanced Multilingual BERT (LS-mBERT), which combines code-switching for lexical alignment with a syntactic graph attention network to encode dependency structure, integrated into mBERT and trained jointly. Across text classification, NER, and semantic parsing tasks, LS-mBERT yields consistent improvements of about $1.0$ to $3.7$ points over strong baselines. The approach also demonstrates generalized cross-lingual transfer benefits and retains competitive gains when paired with stronger bases like XLM-R, highlighting the practical value for multilingual NLP in low-resource languages.

Abstract

Unsupervised cross-lingual transfer involves transferring knowledge between languages without explicit supervision. Although numerous studies have been conducted to improve performance in such tasks by focusing on cross-lingual knowledge, particularly lexical and syntactic knowledge, current approaches are limited as they only incorporate syntactic or lexical information. Since each type of information offers unique advantages and no previous attempts have combined both, we attempt to explore the potential of this approach. In this paper, we present a novel framework called "Lexicon-Syntax Enhanced Multilingual BERT" that combines both lexical and syntactic knowledge. Specifically, we use Multilingual BERT (mBERT) as the base model and employ two techniques to enhance its learning capabilities. The code-switching technique is used to implicitly teach the model lexical alignment information, while a syntactic-based graph attention network is designed to help the model encode syntactic structure. To integrate both types of knowledge, we input code-switched sequences into both the syntactic module and the mBERT base model simultaneously. Our extensive experimental results demonstrate this framework can consistently outperform all baselines of zero-shot cross-lingual transfer, with the gains of 1.0~3.7 points on text classification, named entity recognition (ner), and semantic parsing tasks. Keywords:cross-lingual transfer, lexicon, syntax, code-switching, graph attention network

Figures (7)

• Figure 1: An overview of lexicon-syntax enhanced multilingual BERT ("LS-mBERT"). An example sentence is provided to explain how this framework works. To introduce lexical alignment knowledge, we utilize bilingual dictionaries to randomly replace some words in the sentence with the equivalent words from other languages (pink for German, green for Spanish, light blue for Chinese, and orange for French). Then, an graph attention network (GAT) is developed to encode the syntactic structure of this sentence. The output representation of GAT is sent to the attention heads in multilingual BERT for guiding them to focus on the language-specific structures.
• Figure 2: The architecture of graph attention network ahmad2021syntaxvelivckovic2017graph. Each input token is represented by combining its token embedding and part-of-speech embedding. Each attention head within the graph attention network(GAT) generates a representation for each token embedding by attending to its neighboring tokens in the dependency graph. Next, the resulting representations are concatenated to form the output representation for each token. Finally, we can obtain the representations of the output sequence embeddings from the final layer of GAT.
• Figure 3: Results for generalized zero-shot cross-lingual transfer on mTOP and PAWS-X. We report the performance differences between our method and "mBERT" baseline across all languages.
• Figure 4: Performance improvements for XNLI, PAWS-X, Wikiann, and mTOP across languages. The languages in x-axis are grouped by language families: IE.Germanic (en, de), IE.Romance (es, fr), IE.Slavic (bg, ru), Afro-asiatic (ar), Austro-asiatic (vi), Altaic (tr, ur), IE.Greek (el), IE.Indic (hi), Sino-tibetan (zh), Korean (ko).
• Figure 5: The similarities between languages. We first calculate the centroid representation for each language following libovicky2019language. Then we adopt cosine similarity to evaluate the similarity between English and each target language.