FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression

TL;DR

This work addresses predicting annotator disagreement in multilingual Word-in-Context judgments (DisWiC) for the CoMeDi Shared Task. It combines multilingual sentence embeddings from paraphrase-xlm-r-multilingual-v1 with a four-layer MLP that uses batch normalization and dropout to regress the mean disagreement, trained with AdamW and learning-rate scheduling. The model achieves a competitive 3rd place among 7 teams, with an average Spearman correlation of roughly $\rho=0.124$, and encounters particular challenges with Latin-language data. The results demonstrate the effectiveness of contextual multilingual representations for ordinal disagreement tasks and point to language-specific refinements as a promising avenue for improvement.

Abstract

This paper presents results of our system for CoMeDi Shared Task, focusing on Subtask 2: Disagreement Ranking. Our system leverages sentence embeddings generated by the paraphrase-xlm-r-multilingual-v1 model, combined with a deep neural regression model incorporating batch normalization and dropout for improved generalization. By predicting the mean of pairwise judgment differences between annotators, our method explicitly targets disagreement ranking, diverging from traditional "gold label" aggregation approaches. We optimized our system with a customized architecture and training procedure, achieving competitive performance in Spearman correlation against mean disagreement labels. Our results highlight the importance of robust embeddings, effective model architecture, and careful handling of judgment differences for ranking disagreement in multilingual contexts. These findings provide insights into the use of contextualized representations for ordinal judgment tasks and open avenues for further refinement of disagreement prediction models.

Figures (3)

• Figure 1: The structure of Deep Regression model.
• Figure 2: Structure of BERT and XLM-RoBERTa.
• Figure 3: Training and validation loss while training.