Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classification

Sachin Yadav, Dominik Schlechtweg

TL;DR

Word-in-Context tasks span binary, ordinal, and graded similarity formulations. XL-DURel finetunes a multilingual Sentence Transformer with ranking-oriented losses (AnglE, CoSENT) and target-word marking to exploit ordinal structure in OGWiC data, while training on a merged mix of ordinal CoMeDi and binary WiC datasets. Ranking losses, especially AnglE, achieve top performance on ordinal evaluation and competitive results on binary WiC, with mixing ordinal and binary data further boosting performance. The work showcases a unified, efficient framework for multilingual WiC modeling and provides a practical embedder for word meaning in context, with implications for broader downstream language understanding tasks.

Abstract

We propose XL-DURel, a finetuned, multilingual Sentence Transformer model optimized for ordinal Word-in-Context classification. We test several loss functions for regression and ranking tasks managing to outperform previous models on ordinal and binary data with a ranking objective based on angular distance in complex space. We further show that binary WiC can be treated as a special case of ordinal WiC and that optimizing models for the general ordinal task improves performance on the more specific binary task. This paves the way for a unified treatment of WiC modeling across different task formulations.

XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classification

TL;DR

Word-in-Context tasks span binary, ordinal, and graded similarity formulations. XL-DURel finetunes a multilingual Sentence Transformer with ranking-oriented losses (AnglE, CoSENT) and target-word marking to exploit ordinal structure in OGWiC data, while training on a merged mix of ordinal CoMeDi and binary WiC datasets. Ranking losses, especially AnglE, achieve top performance on ordinal evaluation and competitive results on binary WiC, with mixing ordinal and binary data further boosting performance. The work showcases a unified, efficient framework for multilingual WiC modeling and provides a practical embedder for word meaning in context, with implications for broader downstream language understanding tasks.

Abstract

We propose XL-DURel, a finetuned, multilingual Sentence Transformer model optimized for ordinal Word-in-Context classification. We test several loss functions for regression and ranking tasks managing to outperform previous models on ordinal and binary data with a ranking objective based on angular distance in complex space. We further show that binary WiC can be treated as a special case of ordinal WiC and that optimizing models for the general ordinal task improves performance on the more specific binary task. This paves the way for a unified treatment of WiC modeling across different task formulations.

Paper Structure

This paper contains 36 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. WiC Task
  4. GWiC Task
  5. OGWiC Task
  6. Data
  7. Ordinal WiC
  8. Binary WiC
  9. XL-WiC
  10. MCL-WiC
  11. AM2iCo
  12. Label Mapping
  13. Model
  14. Target Word Marking
  15. Loss Functions
  16. ...and 21 more sections

Figures (3)

  • Figure 1: SBERT architecture at inference.
  • Figure 2: Model evaluation with Krippendorff's $\alpha$ on binary and ordinal test data. CL = Contrastive Loss, COS = Cosine Similarity Loss, CSENT = CoSENT Loss, AnglE = AnglE Loss, XL-LXM = XL-LEXEME, DMistake = DeepMistake.
  • Figure 3: Model evaluation with Spearman's $\rho$ on binary and ordinal test data. CL = Contrastive Loss, COS = Cosine Similarity Loss, CSENT = CoSENT Loss, AnglE = AnglE Loss, XL-LXM = XL-LEXEME.