Toward Informal Language Processing: Knowledge of Slang in Large Language Models

TL;DR

This work addresses the challenge of informal language processing by evaluating large language models' knowledge of slang and by introducing a public benchmark, OpenSub-Slang, derived from movie subtitles. It systematically probes LLMs for slang detection, regional/historical slang source identification, and semantic evaluation, using both behavioral probing and edge probing techniques. The study demonstrates that while GPT-4 and fine-tuned GPT-3.5 outperform BERT-like models on slang tasks, much of the slang knowledge appears to come from exposure rather than a structured semantic representation. The OpenSub-Slang dataset, with usage contexts, regional metadata, and paraphrases, provides a high-quality resource for evaluation and finetuning, enabling better informal language processing while highlighting privacy considerations related to demographic inference from slang.

Abstract

Recent advancement in large language models (LLMs) has offered a strong potential for natural language systems to process informal language. A representative form of informal language is slang, used commonly in daily conversations and online social media. To date, slang has not been comprehensively evaluated in LLMs due partly to the absence of a carefully designed and publicly accessible benchmark. Using movie subtitles, we construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences. We also show how our dataset can be used to probe the output distributions of LLMs for interpretive insights. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance. Furthermore, we show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve substantially better performance than strong zero-shot baselines. Our work offers a comprehensive evaluation and a high-quality benchmark on English slang based on the OpenSubtitles corpus, serving both as a publicly accessible resource and a platform for applying tools for informal language processing.

Figures (8)

• Figure 1: Overview of tasks used to probe knowledge of slang in LLMs.
• Figure 2: Slang detection performance by region.
• Figure 3: Significance level of the regional discrepancies in slang detection performance at both the word-level and the sentence-level. We perform a one-sided test to evaluate whether detection performance on UK slang is indeed significantly better than that of US slang.
• Figure 4: Classification performance on slang source identification tasks.
• Figure 5: Likelihood ratios between samples of corresponding slang and literal tokens.