Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation

Thanet Markchom, Tong Wu, Liting Huang, Huizhi Liang

TL;DR

This work tackles multilingual multimodal idiomaticity representation for image ranking in SemEval-2025 Task 1 Subtask A by combining generative LLMs to produce idiomatic meanings with multilingual CLIP embeddings. It introduces an LLM-based idiom detector, an embedding pipeline that uses either literal compounds or generated idioms, and an ensemble of LLMs to stabilize predictions; a contrastive learning stage with data augmentation further refines embeddings, though fine-tuning yields mixed gains. Experimental results show that multimodal representations outperform baselines that rely solely on nominal compounds, with language-specific best-performing configurations emerging (e.g., GPT-4 plus XLM-R LABSE for English, GPT-3.5 with LABSE-14 for Portuguese). The findings highlight the potential of generative and multimodal signals for idiomaticity tasks and indicate that data-efficient fine-tuning remains a challenge; the work provides code for reproducibility at https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL.

Abstract

SemEval-2025 Task 1 focuses on ranking images based on their alignment with a given nominal compound that may carry idiomatic meaning in both English and Brazilian Portuguese. To address this challenge, this work uses generative large language models (LLMs) and multilingual CLIP models to enhance idiomatic compound representations. LLMs generate idiomatic meanings for potentially idiomatic compounds, enriching their semantic interpretation. These meanings are then encoded using multilingual CLIP models, serving as representations for image ranking. Contrastive learning and data augmentation techniques are applied to fine-tune these embeddings for improved performance. Experimental results show that multimodal representations extracted through this method outperformed those based solely on the original nominal compounds. The fine-tuning approach shows promising outcomes but is less effective than using embeddings without fine-tuning. The source code used in this paper is available at https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL.

UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation

TL;DR

This work tackles multilingual multimodal idiomaticity representation for image ranking in SemEval-2025 Task 1 Subtask A by combining generative LLMs to produce idiomatic meanings with multilingual CLIP embeddings. It introduces an LLM-based idiom detector, an embedding pipeline that uses either literal compounds or generated idioms, and an ensemble of LLMs to stabilize predictions; a contrastive learning stage with data augmentation further refines embeddings, though fine-tuning yields mixed gains. Experimental results show that multimodal representations outperform baselines that rely solely on nominal compounds, with language-specific best-performing configurations emerging (e.g., GPT-4 plus XLM-R LABSE for English, GPT-3.5 with LABSE-14 for Portuguese). The findings highlight the potential of generative and multimodal signals for idiomaticity tasks and indicate that data-efficient fine-tuning remains a challenge; the work provides code for reproducibility at https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL.

Abstract

SemEval-2025 Task 1 focuses on ranking images based on their alignment with a given nominal compound that may carry idiomatic meaning in both English and Brazilian Portuguese. To address this challenge, this work uses generative large language models (LLMs) and multilingual CLIP models to enhance idiomatic compound representations. LLMs generate idiomatic meanings for potentially idiomatic compounds, enriching their semantic interpretation. These meanings are then encoded using multilingual CLIP models, serving as representations for image ranking. Contrastive learning and data augmentation techniques are applied to fine-tune these embeddings for improved performance. Experimental results show that multimodal representations extracted through this method outperformed those based solely on the original nominal compounds. The fine-tuning approach shows promising outcomes but is less effective than using embeddings without fine-tuning. The source code used in this paper is available at https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL.

Paper Structure

This paper contains 19 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Idiomatic Meaning Generation
  4. Embedding Extraction and Image Ranking
  5. Ensemble Method
  6. Fine-Tuning with Contrastive Learning
  7. Data Augmentation
  8. Contrastive Learning Model
  9. Experimental Setup
  10. Datasets
  11. Hyperparameter Settings
  12. Evaluation Metrics
  13. Results and Discussion
  14. Compound Type Detection Results
  15. Image Ranking Results
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overview of the proposed method: An LLM determines whether a compound is idiomatic or literal based on its context sentence. If idiomatic, the LLM generates its idiomatic meaning. A CLIP model then extracts embeddings of the original compound (if literal) or the generated meaning (if idiomatic), along with image and caption embeddings. Finally, cosine similarity is used to compute the ranking score.
  • Figure 2: Training loss, validation loss, and test accuracy, during the fine-tuning of embeddings obtained using LABSE-14, with GPT-3.5, GPT-4, and GPT-4o used for idiomatic meaning generation.
  • Figure 3: Top-1 accuracy on the test set during the fine-tuning of embeddings obtained using LABSE-14, with GPT-3.5 used for idiomatic meaning generation.