Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals
Pouya Shaeri, Ryan T. Woo, Yasaman Mohammadpour, Ariane Middel
TL;DR
This work tackles segmentation brittleness by reframing human corrections as interventional signals rather than passive labels, enabling counterfactual learning and cross-image propagation of fixes. The proposed framework combines a model-agnostic segmentation backbone, an explainable critic interface, and a propagation mechanism that generalizes corrections to visually similar images, using losses that enforce correction-consistency and cross-image transfer: $\\mathcal{L}_{total} = \\mathcal{L}_{seg} + \\lambda_{cf} \\mathcal{L}_{cf} + \\lambda_{prop} \\mathcal{L}_{prop}$ with $\\L_{cf} = \\frac{1}{|R|} \\sum_{i \\in R} \\ell(f_\\theta(x)_i, y^*_i)$ and $\\L_{prop} = \\frac{1}{|M|} \\sum_{(i,j) \\in M} \\ell(f_\\theta(x^j)_i, y^*_i)$. Empirically, the method yields up to 9 mIoU-point gains on cubemap data (and 2–3 mIoU on standard benchmarks), reduces annotation effort by 3–4×, and improves robustness to domain shifts and spurious correlations, demonstrating practical value for real-world domains such as urban climate monitoring and autonomous driving.
Abstract
Segmentation models achieve high accuracy on benchmarks but often fail in real-world domains by relying on spurious correlations instead of true object boundaries. We propose a human-in-the-loop interactive framework that enables interventional learning through targeted human corrections of segmentation outputs. Our approach treats human corrections as interventional signals that show when reliance on superficial features (e.g., color or texture) is inappropriate. The system learns from these interventions by propagating correction-informed edits across visually similar images, effectively steering the model toward robust, semantically meaningful features rather than dataset-specific artifacts. Unlike traditional annotation approaches that simply provide more training data, our method explicitly identifies when and why the model fails and then systematically corrects these failure modes across the entire dataset. Through iterative human feedback, the system develops increasingly robust representations that generalize better to novel domains and resist artifactual correlations. We demonstrate that our framework improves segmentation accuracy by up to 9 mIoU points (12-15\% relative improvement) on challenging cubemap data and yields 3-4$\times$ reductions in annotation effort compared to standard retraining, while maintaining competitive performance on benchmark datasets. This work provides a practical framework for researchers and practitioners seeking to build segmentation systems that are accurate, robust to dataset biases, data-efficient, and adaptable to real-world domains such as urban climate monitoring and autonomous driving.