Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Intra and Inter-language Consistency in Embeddings with ICA

Rongzhi Li, Takeru Matsuda, Hitomi Yanaka

TL;DR

This work addresses the interpretability and cross-language stability of semantic axes derived from word embeddings via Independent Component Analysis (ICA). By applying multiple ICA runs and clustering with Icasso, the authors establish intra-language reproducibility, and they use Hyvärinen-style testing to quantify inter-language correspondences across English, Japanese, and Chinese FastText embeddings. The study demonstrates robust, interpretable semantic axes and identifies a subset of shared axes across languages, supporting the existence of universal semantics and enabling compositional maps for multilingual NLP. The results provide a statistical framework for reliability and universality of semantic axes, with implications for cross-lingual translation and multilingual language understanding.

Abstract

Word embeddings represent words as multidimensional real vectors, facilitating data analysis and processing, but are often challenging to interpret. Independent Component Analysis (ICA) creates clearer semantic axes by identifying independent key features. Previous research has shown ICA's potential to reveal universal semantic axes across languages. However, it lacked verification of the consistency of independent components within and across languages. We investigated the consistency of semantic axes in two ways: both within a single language and across multiple languages. We first probed into intra-language consistency, focusing on the reproducibility of axes by performing ICA multiple times and clustering the outcomes. Then, we statistically examined inter-language consistency by verifying those axes' correspondences using statistical tests. We newly applied statistical methods to establish a robust framework that ensures the reliability and universality of semantic axes.

Exploring Intra and Inter-language Consistency in Embeddings with ICA

TL;DR

This work addresses the interpretability and cross-language stability of semantic axes derived from word embeddings via Independent Component Analysis (ICA). By applying multiple ICA runs and clustering with Icasso, the authors establish intra-language reproducibility, and they use Hyvärinen-style testing to quantify inter-language correspondences across English, Japanese, and Chinese FastText embeddings. The study demonstrates robust, interpretable semantic axes and identifies a subset of shared axes across languages, supporting the existence of universal semantics and enabling compositional maps for multilingual NLP. The results provide a statistical framework for reliability and universality of semantic axes, with implications for cross-lingual translation and multilingual language understanding.

Abstract

Word embeddings represent words as multidimensional real vectors, facilitating data analysis and processing, but are often challenging to interpret. Independent Component Analysis (ICA) creates clearer semantic axes by identifying independent key features. Previous research has shown ICA's potential to reveal universal semantic axes across languages. However, it lacked verification of the consistency of independent components within and across languages. We investigated the consistency of semantic axes in two ways: both within a single language and across multiple languages. We first probed into intra-language consistency, focusing on the reproducibility of axes by performing ICA multiple times and clustering the outcomes. Then, we statistically examined inter-language consistency by verifying those axes' correspondences using statistical tests. We newly applied statistical methods to establish a robust framework that ensures the reliability and universality of semantic axes.
Paper Structure (22 sections, 8 equations, 5 figures, 2 tables)

This paper contains 22 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. ICA
  4. Interpretability in Word Embeddings
  5. Consistency in Embeddings
  6. Intra-language Consistency
  7. Inter-language Consistency
  8. Interpretation of Independent Components
  9. Experimental Settings
  10. Results and Discussion
  11. Overall Results
  12. Quantitative Evaluations
  13. Discussion
  14. Conclusion
  15. Definitions
  16. ...and 7 more sections

Figures (5)

  • Figure 1: An illustration of clustering of independent components within and between languages. The circles represent the clusters created by Icasso, and the numbers indicate their quality indexes. Clusters with high-quality indexes were given interpretations using words. The circles connected by straight lines show components grouped together by checking consistency among languages.
  • Figure 2: Quality index for FastText embeddings.
  • Figure 3: Similarity of Independent Components - English and Japanese.
  • Figure 4: Similarity of Independent Components - English and Chinese.
  • Figure 5: Similarity of Independent Components - Japanese and Chinese.