AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions

TL;DR

This work introduces AnnoABSA, the first web-based annotation tool to support the full spectrum of Aspect-Based Sentiment Analysis (ABSA) tasks, and provides optional Large Language Model-based retrieval-augmented generation suggestions that offer context-aware assistance in a human-in-the-loop approach, keeping the human annotator in control.

Abstract

We introduce AnnoABSA, the first web-based annotation tool to support the full spectrum of Aspect-Based Sentiment Analysis (ABSA) tasks. The tool is highly customizable, enabling flexible configuration of sentiment elements and task-specific requirements. Alongside manual annotation, AnnoABSA provides optional Large Language Model (LLM)-based retrieval-augmented generation (RAG) suggestions that offer context-aware assistance in a human-in-the-loop approach, keeping the human annotator in control. To improve prediction quality over time, the system retrieves the ten most similar examples that are already annotated and adds them as few-shot examples in the prompt, ensuring that suggestions become increasingly accurate as the annotation process progresses. Released as open-source software under the MIT License, AnnoABSA is freely accessible and easily extendable for research and practical applications.

Figures (4)

• Figure 1: UI of AnnoABSA. The UI consists of four components: (1) top navigation bar to change the currently displayed example, (2) annotated text with highlighted sentiment annotations, (3) aspect addition panel, and (4) editable list of added annotations.
• Figure 2: Popup to add or manipulate phrase annotations. In case both aspect term and opinion term annotations are required, the text is displayed twice, once for each phrase annotation. Implicit aspects can be marked using a checkbox.
• Figure 3: F1 score comparison of RAG-based and random sampling approaches. RAG consistently outperforms random sampling across all configurations. Statistical tests (paired t-test or Wilcoxon signed-rank test with Holm-Bonferroni correction) confirm all differences are significant. $M_{diff}$ shows mean performance differences.
• Figure 4: Prompt used for RAG-based suggestion prediction. The prompt includes a task description with explanations of sentiment elements, ten in-context demonstrations, and the target text for aspect prediction. The few-shot examples shown are taken from the Coursera dataset and include annotations for ASQP