NLP Datasets for Idiom and Figurative Language Tasks

TL;DR

This work tackles the challenge of modeling idiom and figurative language in NLP by building large, up-to-date datasets derived from Common Crawl (PIFL-OSCAR) and C4 (PIFL-C4), along with human-annotated, model-agnostic subsets (IFL-OSCAR-A and IFL-C4-A). It introduces a semi-automatic annotation pipeline and a Span Search Algorithm to enrich data with linguistic priors and embedding-based cues, enabling robust evaluation across architectures. Through experiments with BERT-based sequence tagging and zero-shot LLM prompting, the authors show strong in-dataset performance for baselines, but limited cross-dataset generalization for LLMs, while PIFL-OSCAR-based training yields the best generalization in cross-evaluation. The paper provides valuable resources for standardized idiom/figurative-language evaluation and highlights avenues for improving LLM understanding of non-literal language, including multilingual extensions and confidence-aware labeling.

Abstract

Idiomatic and figurative language form a large portion of colloquial speech and writing. With social media, this informal language has become more easily observable to people and trainers of large language models (LLMs) alike. While the advantage of large corpora seems like the solution to all machine learning and Natural Language Processing (NLP) problems, idioms and figurative language continue to elude LLMs. Finetuning approaches are proving to be optimal, but better and larger datasets can help narrow this gap even further. The datasets presented in this paper provide one answer, while offering a diverse set of categories on which to build new models and develop new approaches. A selection of recent idiom and figurative language datasets were used to acquire a combined idiom list, which was used to retrieve context sequences from a large corpus. One large-scale dataset of potential idiomatic and figurative language expressions and two additional human-annotated datasets of definite idiomatic and figurative language expressions were created to evaluate the baseline ability of pre-trained language models in handling figurative meaning through idiom recognition (detection) tasks. The resulting datasets were post-processed for model agnostic training compatibility, utilized in training, and evaluated on slot labeling and sequence tagging.

Figures (10)

• Figure 1: Data retrieval and annotation structure.
• Figure 2: File distribution comparison showing a single file is a suitable sample of the collection.
• Figure 3: "Sometimes" and "always" initial annotation.
• Figure 4: "Figurative" and "literal" annotation example.
• Figure 5: BERT-based model utilizing sequence classification for idiom recognition tasks. The red highlight indicates the idiom phrase in the sentence.