Token-free Models for Sarcasm Detection

TL;DR

Token-free NLP enables processing raw text without explicit tokenization, addressing issues like OOV words and noise. The study benchmarks ByT5-small and CANINE on sarcasm detection across Twitter and News Headlines, comparing against token-based baselines. ByT5-small achieves $89.87\%$ accuracy on News Headlines and CANINE-s achieves $72.88\%$ accuracy on Twitter, with improvements of $0.77\%$ and $0.49\%$, respectively, over token-based baselines, achieving state-of-the-art results. The results illustrate the robustness of token-free models to noisy social media text and their applicability to multilingual, emoji-rich content.

Abstract

Tokenization is a foundational step in most natural language processing (NLP) pipelines, yet it introduces challenges such as vocabulary mismatch and out-of-vocabulary issues. Recent work has shown that models operating directly on raw text at the byte or character level can mitigate these limitations. In this paper, we evaluate two token-free models, ByT5 and CANINE, on the task of sarcasm detection in both social media (Twitter) and non-social media (news headlines) domains. We fine-tune and benchmark these models against token-based baselines and state-of-the-art approaches. Our results show that ByT5-small and CANINE outperform token-based counterparts and achieve new state-of-the-art performance, improving accuracy by 0.77% and 0.49% on the News Headlines and Twitter Sarcasm datasets, respectively. These findings underscore the potential of token-free models for robust NLP in noisy and informal domains such as social media.