Exploring the Performance of Large Language Models on Subjective Span Identification Tasks

TL;DR

The paper evaluates multiple open-source LLM families (Qwen and Llama) on subjective span identification across three NLP tasks (sentiment, offensive language, and claim verification) using instruction-tuning, in-context learning, and chain-of-thought prompting. It compares against a BERT baseline and reports token-level ($TF1$) and span-level ($SF1$) F1 scores, revealing that few-shot and instruction-tuning strategies yield strong performance on complex, interrelated spans, while model size offers only marginal gains. The findings show LLMs excel at explicit, context-independent spans but struggle with nuanced, subjective spans, and that in low-resource settings smaller models with task-tuned supervision can outperform larger LLMs, though few-shot prompts can help LLMs approach baselines. Overall, the work highlights how text structure and prompting strategies influence span extraction and provides guidance for deploying LLMs in explainability-focused NLP tasks. The study also suggests future directions including multilingual and multimodal extensions to broaden applicability.

Abstract

Identifying relevant text spans is important for several downstream tasks in NLP, as it contributes to model explainability. While most span identification approaches rely on relatively smaller pre-trained language models like BERT, a few recent approaches have leveraged the latest generation of Large Language Models (LLMs) for the task. Current work has focused on explicit span identification like Named Entity Recognition (NER), while more subjective span identification with LLMs in tasks like Aspect-based Sentiment Analysis (ABSA) has been underexplored. In this paper, we fill this important gap by presenting an evaluation of the performance of various LLMs on text span identification in three popular tasks, namely sentiment analysis, offensive language identification, and claim verification. We explore several LLM strategies like instruction tuning, in-context learning, and chain of thought. Our results indicate underlying relationships within text aid LLMs in identifying precise text spans.

Figures (7)

• Figure 1: Token F1 and Span F1 scores for the complex text span identification. The plots show the scores for each type of span when extracted individually and combined.
• Figure 2: The prompt template used in our experiments.
• Figure 3: Outputs for the TBO task. The output for each approach is shown. For few-shot, only the 5-shot output is shown.
• Figure 4: Outputs for the ABSA task. The output for each approach is shown. For few-shot, only the 5-shot output is shown.
• Figure 5: Outputs for the CSI task. The output for each approach is shown. For few-shot, only the 5-shot output is shown.