ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation

TL;DR

Araida tackles the limited-data data-annotation problem by integrating an annotation model with a KNN-based analogical reasoning reference. It introduces an error-aware integration strategy that weights the two sources dynamically per example, guided by neighborhood error signals and local density, and optimizes the components via a coordinated loss with differentiable KNN support. Across word- and sentence-level tasks, Araida consistently reduces required human corrections, achieving an average MCA improvement of 11.02% and demonstrating robustness across classic and LLM-based annotation models. The framework is modular and adaptable, improving annotation accuracy while remaining compatible with various datasets, models, and even active-learning setups, thereby enhancing practical annotation efficiency in resource-constrained settings.

Abstract

Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose Araida, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. Araida involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN's predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that Araida is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.

Figures (8)

• Figure 1: Comparison between manual annotation and interactive annotation.
• Figure 2: Example on span relation annotation. An under-trained annotation model results in more suggestion errors and increases human correction effort. Araida improves the model annotation accuracy via the KNN model and the error-aware integration strategy for dynamical coordination of annotations.
• Figure 3: MCA scores using ChatGPT-based methods. We omit the ChatGPT with AL results because we are unable to estimate its prediction uncertainty. Araida can further improve ChatGPT's performance
• Figure 4: Analyzing our integration strategy with the Dist./FT model. The solid lines show the MAC scores of the annotation model $f(\cdot)$, separated by examples with $\lambda > 0.5$ (higher weights assigned to the annotation model $f(\cdot)$) and $\lambda \leq 0.5$ (higher weights assigned to KNN). The dotted line shows KNN's performance on the latter set.
• Figure 5: MAC scores of various methods with synthesized label noise on the SST-5 dataset. Dist./FT is used as the annotation model. Araida-dis refers to Araida with a modified datastore maintenance strategy.