Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Strategies for Span Labeling with Large Language Models

Danil Semin, Ondřej Dušek, Zdeněk Kasner

TL;DR

Grounding in generative LLMs for span labeling is nontrivial because decoder-only models lack explicit references to input spans. The authors categorize span-labeling strategies into tagging, indexing, and matching, and introduce LogitMatch, a constrained decoding method, to enforce valid input-span generation without fine-tuning. Across four tasks, tagging provides robust baseline performance, while LogitMatch mitigates common matching failures and can outperform other strategies in certain setups, with indexing helping mainly when multiple occurrences must be distinguished. The work offers practical guidance for deploying LLM-based span labeling and highlights trade-offs between robustness, token efficiency, reasoning overhead, and model size.

Abstract

Large language models (LLMs) are increasingly used for text analysis tasks, such as named entity recognition or error detection. Unlike encoder-based models, however, generative architectures lack an explicit mechanism to refer to specific parts of their input. This leads to a variety of ad-hoc prompting strategies for span labeling, often with inconsistent results. In this paper, we categorize these strategies into three families: tagging the input text, indexing numerical positions of spans, and matching span content. To address the limitations of content matching, we introduce LogitMatch, a new constrained decoding method that forces the model's output to align with valid input spans. We evaluate all methods across four diverse tasks. We find that while tagging remains a robust baseline, LogitMatch improves upon competitive matching-based methods by eliminating span matching issues and outperforms other strategies in some setups.

Strategies for Span Labeling with Large Language Models

TL;DR

Grounding in generative LLMs for span labeling is nontrivial because decoder-only models lack explicit references to input spans. The authors categorize span-labeling strategies into tagging, indexing, and matching, and introduce LogitMatch, a constrained decoding method, to enforce valid input-span generation without fine-tuning. Across four tasks, tagging provides robust baseline performance, while LogitMatch mitigates common matching failures and can outperform other strategies in certain setups, with indexing helping mainly when multiple occurrences must be distinguished. The work offers practical guidance for deploying LLM-based span labeling and highlights trade-offs between robustness, token efficiency, reasoning overhead, and model size.

Abstract

Large language models (LLMs) are increasingly used for text analysis tasks, such as named entity recognition or error detection. Unlike encoder-based models, however, generative architectures lack an explicit mechanism to refer to specific parts of their input. This leads to a variety of ad-hoc prompting strategies for span labeling, often with inconsistent results. In this paper, we categorize these strategies into three families: tagging the input text, indexing numerical positions of spans, and matching span content. To address the limitations of content matching, we introduce LogitMatch, a new constrained decoding method that forces the model's output to align with valid input spans. We evaluate all methods across four diverse tasks. We find that while tagging remains a robust baseline, LogitMatch improves upon competitive matching-based methods by eliminating span matching issues and outperforms other strategies in some setups.
Paper Structure (55 sections, 2 figures, 9 tables)

This paper contains 55 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Existing Strategies for Span Labeling with Generative Models
  4. Tagging: Marking Spans in the Input Text
  5. Custom Tags
  6. XML-like Tags
  7. BIO Tags
  8. Indexing: Referring to Character Indices
  9. Character-level
  10. Word-Level+Numbering
  11. Matching: String Matching of the Span Content
  12. Unstructured List
  13. JSON Output
  14. Issues with Existing Methods
  15. LLMs cannot copy reliably.
  16. ...and 40 more sections

Figures (2)

  • Figure 1: Illustration of main approaches to span labeling with LLMs, along with the associated issues. The issues stem from the fact that LLMs have no explicit mechanism to ground their outputs in the input text.
  • Figure 2: Error rates for each method (highlighted for each column separately), average for open LLMs across all datasets. We report three kinds of errors: error in parsing the model response, error in matching the span content, and error in matching the category label to one of the existing labels.