Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization Measures for Zero-Shot Cross-Lingual Transfer

Saksham Bassi, Duygu Ataman, Kyunghyun Cho

TL;DR

A novel and stable algorithm is proposed to reliably compute the sharpness of a model optimum, and its correlation with successful cross-lingual transfer is demonstrated.

Abstract

A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language model evaluation tasks lack information metrics about model generalization and their applicability in a new setting is measured using task and language-specific downstream performance, which is often lacking in many languages and tasks. In this paper, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models in cross-lingual zero-shot settings. In addition to traditional measures such as variance in parameters after training and distance from initialization, we also measure the effectiveness of sharpness in loss landscape in capturing the success in cross-lingual transfer and propose a novel and stable algorithm to reliably compute the sharpness of a model optimum that correlates to generalization.

Generalization Measures for Zero-Shot Cross-Lingual Transfer

TL;DR

A novel and stable algorithm is proposed to reliably compute the sharpness of a model optimum, and its correlation with successful cross-lingual transfer is demonstrated.

Abstract

A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language model evaluation tasks lack information metrics about model generalization and their applicability in a new setting is measured using task and language-specific downstream performance, which is often lacking in many languages and tasks. In this paper, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models in cross-lingual zero-shot settings. In addition to traditional measures such as variance in parameters after training and distance from initialization, we also measure the effectiveness of sharpness in loss landscape in capturing the success in cross-lingual transfer and propose a novel and stable algorithm to reliably compute the sharpness of a model optimum that correlates to generalization.
Paper Structure (11 sections, 5 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 11 sections, 5 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Sharpness-based Optimization
  5. Generalization Measures
  6. Experiments
  7. Data, Model details, and Settings
  8. Conclusion
  9. Appendix
  10. Visualization results
  11. Additional experiments

Figures (7)

  • Figure 1: Scatter plot of Margin values and Sharpness ($\phi_{\text{difference}}$) values for each mBERT model (on XNLI dataset) with different objectives language-wise to show the relationship between sharpness and generalization.
  • Figure 2: Scatter plot of difference-based sharpness measure with test performance for all models combined.
  • Figure 3: Scatter plots of margin of individual models and their corresponding performance on test set language-wise on XNLI dataset.
  • Figure 4: Scatter plots of the proposed difference-based sharpness ($\phi_{\text{difference}}$) of individual models and their corresponding performance on test set language-wise on XNLI Dataset.
  • Figure 5: Scatter plot of Frobenius distance from initialization and Test accuracy for each model type (trained multiple times independenty).
  • ...and 2 more figures