Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review

TL;DR

The paper addresses the bottleneck of annotating large image datasets for deep learning by surveying AI-assisted image annotation systems that generate textual hints and descriptions. It analyzes how deep learning, self-supervised learning, active learning, and multi-modal and neuro-symbolic approaches can produce text outputs to guide annotators across CV tasks. Key contributions include a taxonomy of assistive system types, evaluation metrics, commonly used datasets, and a critical assessment of the scarcity of public-text-output annotation tools, with examples and limitations. The work highlights future directions toward cross-modal alignment, large-language-model integration, and collaborative data-sharing to enable efficient, explainable AI-assisted annotation.

Abstract

While supervised learning has achieved significant success in computer vision tasks, acquiring high-quality annotated data remains a bottleneck. This paper explores both scholarly and non-scholarly works in AI-assistive deep learning image annotation systems that provide textual suggestions, captions, or descriptions of the input image to the annotator. This potentially results in higher annotation efficiency and quality. Our exploration covers annotation for a range of computer vision tasks including image classification, object detection, regression, instance, semantic segmentation, and pose estimation. We review various datasets and how they contribute to the training and evaluation of AI-assistive annotation systems. We also examine methods leveraging neuro-symbolic learning, deep active learning, and self-supervised learning algorithms that enable semantic image understanding and generate free-text output. These include image captioning, visual question answering, and multi-modal reasoning. Despite the promising potential, there is limited publicly available work on AI-assistive image annotation with textual output capabilities. We conclude by suggesting future research directions to advance this field, emphasizing the need for more publicly accessible datasets and collaborative efforts between academia and industry.

Figures (5)

• Figure 1: An overview of an AI-assisted image annotation system. The system begins with unlabeled image training data which is processed through various blocks. The Vision-to-Text Block utilizes image captioning, VQA, and multi-modal alignment to provide predictions. The Pretrained Vision Task Block handles image segmentation, pose estimation, and one-shot classification to generate vision task predictions. The Semantic Image Search Block uses self-supervised learning and active learning to assist the annotator in semantic search. Human annotators receive textual and visual suggestions to annotate the images, which are then used to fine-tune the vision task and semantic search models. The final interface allows annotators to accept, edit, or show similar annotations.
• Figure 2: [Source: https://towardsdatascience.com/object-detection-with-convolutional-neural-networks-c9d729eedc18]: An image depicting image classification with 2 classes, a cat and a dog. The predictions are in the form of probabilities that are then mapped to the class labels based on the highest probability.
• Figure 3: [Source: https://www.hackerearth.com/blog/developers/introduction-to-object-detection/] results on a city street showing multiple instances of humans, statues, and lights. The bounding boxes for each object are regression prediction outputs for the rectangular coordinates around each region of interest
• Figure 4: [Source: https://keymakr.com/blog/instance-vs-semantic-segmentation/] results of multiple cars detected and segmented at different distances from the viewpoint. Segmentation models typically provide bounding boxes and class labels for each detected object. Predictions usually consist of $K$ binary masks of size $n \times m$, outlining the pixel locations for all the $K$ detected objects in $n \times m$ image.
• Figure 5: [Source: Singh:HumanPoseEstimation19Example Pose Estimation] annotations with the head, neck, shoulders, elbows, hips, knees, and ankles as key-points