Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

TL;DR

The paper tackles the high cost of creating fine-grained opinion datasets by using Large Language Models as automatic annotators for ASTE and ACOS, paired with a declarative DSPy-driven pipeline and an LLM-based adjudicator to resolve label disagreements. It evaluates three LLM sizes on laptop and restaurant domains without fine-tuning, using in-context learning with optimized prompts, and reports both standard accuracy and Krippendorff's alpha to assess alignment and reliability. Results show that larger models improve annotation quality, and the adjudication step yields the strongest gains for ASTE and ACOS in many settings, though ACOS remains more challenging due to implicit aspects and domain variance. Overall, the approach demonstrates a scalable, cost-effective pathway to generate high-quality fine-grained opinion datasets, enabling broader domain coverage and potentially accelerating downstream ABSA research and deployment.

Abstract

Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments, including the addressed entity. Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications. We explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis, addressing the shortage of domain-specific labelled datasets. In this work, we use a declarative annotation pipeline. This approach reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a novel methodology for an LLM to adjudicate multiple labels and produce final annotations. After trialling the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks, we show that LLMs can serve as automatic annotators and adjudicators, achieving high Inter-Annotator Agreement across individual LLM-based annotators. This reduces the cost and human effort needed to create these fine-grained opinion-annotated datasets.

Figures (2)

• Figure 1: LLM-Based Annotation Pipeline using DSPy (Left) with LLM-as-adjudicator (Right)
• Figure :