Feedback-driven object detection and iterative model improvement

TL;DR

This paper tackles the annotation bottleneck in building high-quality object-detection datasets by introducing a lightweight, open-source platform that couples semi-automatic labeling with iterative model refinement. It uses a pre-trained SSD300 on COCO to generate initial labels, which humans then correct to progressively improve the model through snapshots and fine-tuning. Quantitatively, the approach achieves up to $53.82\%$ reduction in annotation time while maintaining or improving accuracy (IoU and F1), demonstrated on an agricultural dataset of 200 images. The work highlights the practical value of human-in-the-loop labeling for scalable dataset creation and provides guidance for developing future annotation systems.

Abstract

Automated object detection has become increasingly valuable across diverse applications, yet efficient, high-quality annotation remains a persistent challenge. In this paper, we present the development and evaluation of a platform designed to interactively improve object detection models. The platform allows uploading and annotating images as well as fine-tuning object detection models. Users can then manually review and refine annotations, further creating improved snapshots that are used for automatic object detection on subsequent image uploads - a process we refer to as semi-automatic annotation resulting in a significant gain in annotation efficiency. Whereas iterative refinement of model results to speed up annotation has become common practice, we are the first to quantitatively evaluate its benefits with respect to time, effort, and interaction savings. Our experimental results show clear evidence for a significant time reduction of up to 53% for semi-automatic compared to manual annotation. Importantly, these efficiency gains did not compromise annotation quality, while matching or occasionally even exceeding the accuracy of manual annotations. These findings demonstrate the potential of our lightweight annotation platform for creating high-quality object detection datasets and provide best practices to guide future development of annotation platforms. The platform is open-source, with the frontend and backend repositories available on GitHub. To support the understanding of our labeling process, we have created an explanatory video demonstrating the methodology using microscopy images of E. coli bacteria as an example.

Figures (6)

• Figure 1: Illustration of the iterative annotation workflow in our platform. The process begins with a pre-trained object detection model predicting bounding boxes on unlabeled images. Users then correct these predictions, and the refined annotations are fed back into the model for incremental improvement. This feedback loop progressively enhances model accuracy, reducing manual annotation effort over time.
• Figure 2: The user interface (UI) of the annotation platform. The left pane shows the project management interface, where users can manage projects, view uploaded image bundles, and monitor the status of model fine-tuning. The right pane displays the annotation editor, where users can review and adjust predicted bounding boxes, creating high-quality annotations through a semi-automatic workflow.
• Figure 3: Example images from the agricultural dataset dt-mars, captured by an autonomous agricultural robot. These images illustrate some of the challenging conditions encountered during the annotation process, including varying camera distances, shadows, and partial occlusion of plants.
• Figure 4: Comparison of the average annotation time per bundle (in seconds) between manual and semi-automatic processes. The results show a significant time reduction using the semi-automatic approach, particularly in later bundles as the model's accuracy improves, reducing the need for manual intervention.
• Figure 5: User interaction time metrics comparing the time required for creating, adjusting, or removing bounding boxes in manual and semi-automatic workflows.