Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distilling Monolingual and Crosslingual Word-in-Context Representations

Yuki Arase, Tomoyuki Kajiwara

TL;DR

The paper tackles distilling word meaning in context from frozen pre-trained masked language models without relying on labeled data or fine-tuning. It introduces an autoencoder-based distiller that fuses information from multiple hidden layers via self-attention to produce word-in-context representations, with separate meaning and context components and a cross-reconstruction training objective. Leveraging automatically generated corpora from round-trip translations, word alignment, and masked token prediction, the method supports both monolingual and crosslingual settings and preserves contextual richness better than transformation-based approaches. Empirical results show competitive monolingual performance, especially for STS, and notable crosslingual gains on MCL-WiC and related tasks, with ablations confirming the necessity of negative samples for effective distillation. The work has practical impact for deploying robust, language-flexible lexical representations without expensive annotation or continual model updates, enabling better crosslingual similarity estimation and downstream semantic tasks.

Abstract

In this study, we propose a method that distils representations of word meaning in context from a pre-trained masked language model in both monolingual and crosslingual settings. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. Different from existing approaches, our method does not require human-annotated corpora nor updates of the parameters of the pre-trained model. The latter feature is appealing for practical scenarios where the off-the-shelf pre-trained model is a common asset among different applications. Specifically, our method learns to combine the outputs of different hidden layers of the pre-trained model using self-attention. Our auto-encoder based training only requires an automatically generated corpus. To evaluate the performance of the proposed approach, we performed extensive experiments using various benchmark tasks. The results on the monolingual tasks confirmed that our representations exhibited a competitive performance compared to that of the previous study for the context-aware lexical semantic tasks and outperformed it for STS estimation. The results of the crosslingual tasks revealed that the proposed method largely improved crosslingual word representations of multilingual pre-trained models.

Distilling Monolingual and Crosslingual Word-in-Context Representations

TL;DR

The paper tackles distilling word meaning in context from frozen pre-trained masked language models without relying on labeled data or fine-tuning. It introduces an autoencoder-based distiller that fuses information from multiple hidden layers via self-attention to produce word-in-context representations, with separate meaning and context components and a cross-reconstruction training objective. Leveraging automatically generated corpora from round-trip translations, word alignment, and masked token prediction, the method supports both monolingual and crosslingual settings and preserves contextual richness better than transformation-based approaches. Empirical results show competitive monolingual performance, especially for STS, and notable crosslingual gains on MCL-WiC and related tasks, with ablations confirming the necessity of negative samples for effective distillation. The work has practical impact for deploying robust, language-flexible lexical representations without expensive annotation or continual model updates, enabling better crosslingual similarity estimation and downstream semantic tasks.

Abstract

In this study, we propose a method that distils representations of word meaning in context from a pre-trained masked language model in both monolingual and crosslingual settings. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. Different from existing approaches, our method does not require human-annotated corpora nor updates of the parameters of the pre-trained model. The latter feature is appealing for practical scenarios where the off-the-shelf pre-trained model is a common asset among different applications. Specifically, our method learns to combine the outputs of different hidden layers of the pre-trained model using self-attention. Our auto-encoder based training only requires an automatically generated corpus. To evaluate the performance of the proposed approach, we performed extensive experiments using various benchmark tasks. The results on the monolingual tasks confirmed that our representations exhibited a competitive performance compared to that of the previous study for the context-aware lexical semantic tasks and outperformed it for STS estimation. The results of the crosslingual tasks revealed that the proposed method largely improved crosslingual word representations of multilingual pre-trained models.
Paper Structure (41 sections, 10 equations, 4 figures, 11 tables, 1 algorithm)

This paper contains 41 sections, 10 equations, 4 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Representation Learning for Word Meaning in Context
  4. Representation Disentanglement
  5. Distilling Word Meaning in Context
  6. Learning Framework
  7. Cross Reconstruction
  8. Training Corpus Creation
  9. Round-trip Translation and Word Alignment
  10. Masked Token Prediction
  11. Training for Crosslingual Scenario
  12. Experimental Setup
  13. Context-aware Lexical Semantic Tasks
  14. Within-word Tasks
  15. Inter-word Tasks
  16. ...and 26 more sections

Figures (4)

  • Figure 1: Distillation of word meaning in context via an autoencoder
  • Figure 2: Cross reconstruction with automatically generated positive and negative samples (the orange arrows indicate differences between the monolingual and crosslingual settings).
  • Figure 3: Distribution of cosine similarities between monolingual meaning/context representations and BERT-Large layers computed on the SCWS corpus. The darker colours indicate higher similarities.
  • Figure 4: Distribution of cosine similarities between crosslingual meaning/context representations and mSBERT layers computed on the MCL-WiC en-ru corpus. The darker colours indicate higher similarities.