Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

Tong Wu, Thanet Markchom, Huizhi Liang

TL;DR

This paper systematically compares three approaches to large language model prompting with structured reasoning and explicit decision rules and reveals that structured prompting with decision rules substantially outperforms both fine-tuned models and embedding-based approaches and that prompt design matters more than model scale for this task.

Abstract

Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1--5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules substantially outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task. The code is publicly available at https://github.com/tongwu17/SemEval-2026-Task5.

NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

TL;DR

This paper systematically compares three approaches to large language model prompting with structured reasoning and explicit decision rules and reveals that structured prompting with decision rules substantially outperforms both fine-tuned models and embedding-based approaches and that prompt design matters more than model scale for this task.

Abstract

Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1--5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules substantially outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task. The code is publicly available at https://github.com/tongwu17/SemEval-2026-Task5.
Paper Structure (38 sections, 3 figures, 3 tables)

This paper contains 38 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Task Description.
  4. Word Sense Disambiguation.
  5. Transformer Fine-Tuning.
  6. Sentence Embeddings.
  7. LLM Prompting.
  8. Methodology
  9. Embedding-Based Methods
  10. MPNet + Ridge Regression.
  11. RoBERTa + XGBoost.
  12. Transformer Fine-Tuning
  13. ELECTRA-base and ELECTRA-large + LoRA.
  14. DeBERTa-large + LoRA with Pairwise and Uncertainty Losses.
  15. LLM Prompting
  16. ...and 23 more sections

Figures (3)

  • Figure 1: System overview of the three approaches. All approaches take the same input (narrative story, homonym, candidate word sense) and output a plausibility rating from 1 to 5.
  • Figure 2: Error analysis of GPT-5.2 (P1) on the test set. (a) MAE across groups divided by gold rating. (b) Distribution of gold average scores vs. predicted scores; predictions cluster at discrete integer values.
  • Figure 3: Error analysis of GPT-4o (P2) on the test set. (a) MAE across groups divided by gold rating. (b) Distribution of gold average scores vs. predicted scores; predictions cluster at discrete integer values.