Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives

TL;DR

The paper investigates modeling annotator disagreement in Word-in-Context (WiC) tasks within the CoMeDi shared task, focusing on how contextual meaning relates to judgment variability. It proposes three methods—feature enrichment of contextual embeddings, Adapter-based task-specific representations, and ensemble classifiers/regressors—built on XLM-RoBERTa to predict ordinal median judgments and disagreement magnitudes in OGWiC and DisWiC, evaluated with $\alpha$ and $\rho$. Results show that enriched features and adapters yield improvements, with the Adapter-based approach performing best for OGWiC and mixed results for DisWiC; overall, the methods are competitive with official baselines and top submissions in different subtasks. The work advances understanding of context-driven disagreement in multilingual settings and provides public code to foster further research and method-combining in this area.

Abstract

In this work, we evaluate annotator disagreement in Word-in-Context (WiC) tasks exploring the relationship between contextual meaning and disagreement as part of the CoMeDi shared task competition. While prior studies have modeled disagreement by analyzing annotator attributes with single-sentence inputs, this shared task incorporates WiC to bridge the gap between sentence-level semantic representation and annotator judgment variability. We describe three different methods that we developed for the shared task, including a feature enrichment approach that combines concatenation, element-wise differences, products, and cosine similarity, Euclidean and Manhattan distances to extend contextual embedding representations, a transformation by Adapter blocks to obtain task-specific representations of contextual embeddings, and classifiers of varying complexities, including ensembles. The comparison of our methods demonstrates improved performance for methods that include enriched and task-specfic features. While the performance of our method falls short in comparison to the best system in subtask 1 (OGWiC), it is competitive to the official evaluation results in subtask 2 (DisWiC).

Figures (1)

• Figure 1: Densities of cosine similarity (x-axis) of context embeddings $e_1$ and $e_2$ vs median similarity rating (y-axis). Note that the x-axis does not start at 0.