HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains

TL;DR

HEPHA tackles the challenge of labeling images in specialized domains where labeled data and expert time are scarce. It combines inductive logic programming with a visual programming interface to generate and refine interpretable labeling rules from a small set of labeled images, applying them to unlabeled data. Three recommendation mechanisms—holdout accuracy guidance, saliency-based predicate refinement, and multi-criteria active learning—guide rule and image selections, and a model-agnostic predicate extractor makes the system adaptable to various tasks. A within-subjects user study shows HEPHA achieves high accuracy (around 87–91%) with less labeling and competitive or faster labeling times than a ResNet baseline, with users favoring rule editing and transparency features. The results suggest that eliciting domain knowledge through interpretable, rule-based automation can substantially improve labeling efficiency and accuracy in specialized settings.

Abstract

Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks. Our code is available at https://github.com/Neural-Symbolic-Image-Labeling/NSILWeb.

Figures (15)

• Figure 1: Overview of HEPHA system architecture.
• Figure 2: The image gallery of HEPHA. Users are allowed to view the extracted visual information when viewing images individually. Images that are recommended by active learning are highlighted and prioritized in the gallery.
• Figure 3: Rule accuracy and labeling progress statistics visualized by HEPHA.
• Figure 4: The interface of the rule editor in HEPHA. Users are able to add different types of predicates and clause operators via drag-and-drop (A). Users can select objects from the drop-down list, which they are ranked by a saliency-based method (B).
• Figure 5: Labeling Rule Example 1: Classify indoor scene images as a conference room.