Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bayesian Multilingual Document Model for Zero-shot Topic Identification and Discovery

Santosh Kesiraju, Sangeet Sagar, Ondřej Glembek, Lukáš Burget, Ján Černocký, Suryakanth V Gangashetty

TL;DR

The model is an extension of BaySMM that learns to represent the document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance through linear classifiers that benefit zero-shot cross-lingual topic identification.

Abstract

In this paper, we present a Bayesian multilingual document model for learning language-independent document embeddings. The model is an extension of BaySMM [Kesiraju et al 2020] to the multilingual scenario. It learns to represent the document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance. We propagate the learned uncertainties through linear classifiers that benefit zero-shot cross-lingual topic identification. Our experiments on 17 languages show that the proposed multilingual Bayesian document model performs competitively, when compared to other systems based on large-scale neural networks (LASER, XLM-R, mUSE) on 8 high-resource languages, and outperforms these systems on 9 mid-resource languages. We revisit cross-lingual topic identification in zero-shot settings by taking a deeper dive into current datasets, baseline systems and the languages covered. We identify shortcomings in the existing evaluation protocol (MLDoc dataset), and propose a robust alternative scheme, while also extending the cross-lingual experimental setup to 17 languages. Finally, we consolidate the observations from all our experiments, and discuss points that can potentially benefit the future research works in applications relying on cross-lingual transfers.

A Bayesian Multilingual Document Model for Zero-shot Topic Identification and Discovery

TL;DR

The model is an extension of BaySMM that learns to represent the document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance through linear classifiers that benefit zero-shot cross-lingual topic identification.

Abstract

In this paper, we present a Bayesian multilingual document model for learning language-independent document embeddings. The model is an extension of BaySMM [Kesiraju et al 2020] to the multilingual scenario. It learns to represent the document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance. We propagate the learned uncertainties through linear classifiers that benefit zero-shot cross-lingual topic identification. Our experiments on 17 languages show that the proposed multilingual Bayesian document model performs competitively, when compared to other systems based on large-scale neural networks (LASER, XLM-R, mUSE) on 8 high-resource languages, and outperforms these systems on 9 mid-resource languages. We revisit cross-lingual topic identification in zero-shot settings by taking a deeper dive into current datasets, baseline systems and the languages covered. We identify shortcomings in the existing evaluation protocol (MLDoc dataset), and propose a robust alternative scheme, while also extending the cross-lingual experimental setup to 17 languages. Finally, we consolidate the observations from all our experiments, and discuss points that can potentially benefit the future research works in applications relying on cross-lingual transfers.

Paper Structure

This paper contains 25 sections, 9 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Training multilingual models
  3. Contributions of the paper
  4. MBay: Multilingual Bayesian Model
  5. Variational Bayes training
  6. Extending to newer languages
  7. Inferring embeddings
  8. Classification exploiting uncertainties
  9. Generative classifier
  10. Discriminative classifier
  11. Experimental setup
  12. Data for multilingual training
  13. Dataset preparation for topic ID
  14. MBay configurations
  15. Topic ID systems for MBay
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Graphical representation of the proposed multilingual Bayesian model, where $L$ represents number of languages and $D$ denotes number of $L$-way parallel documents (translations). $\{\mathbf{m}^{\mkern-1.2mu(\ell)}, \mathbf{T}^{\mkern-1.2mu(\ell)}\}\,\forall \ell$ are document-independent, language-specific model parameters, whereas $\mathbf{w}_d$ is document-specific but language-independent random variable (embedding), and $\mathbf{x}^{\mkern-1.2mu(\ell)}_d$ is the observed vector of word counts representing document $d$ from language $\ell$.