Dirichlet-Based Coarse-to-Fine Example Selection For Open-Set Annotation

TL;DR

This paper introduces simplex-based evidential deep learning (EDL) to break translation invariance and distinguish known and unknown classes by considering evidence-based data and distribution uncertainty simultaneously and proposes a Dirichlet-based Coarse-to-Fine Example Selection (DCFS) strategy accordingly.

Abstract

Active learning (AL) has achieved great success by selecting the most valuable examples from unlabeled data. However, they usually deteriorate in real scenarios where open-set noise gets involved, which is studied as open-set annotation (OSA). In this paper, we owe the deterioration to the unreliable predictions arising from softmax-based translation invariance and propose a Dirichlet-based Coarse-to-Fine Example Selection (DCFS) strategy accordingly. Our method introduces simplex-based evidential deep learning (EDL) to break translation invariance and distinguish known and unknown classes by considering evidence-based data and distribution uncertainty simultaneously. Furthermore, hard known-class examples are identified by model discrepancy generated from two classifier heads, where we amplify and alleviate the model discrepancy respectively for unknown and known classes. Finally, we combine the discrepancy with uncertainties to form a two-stage strategy, selecting the most informative examples from known classes. Extensive experiments on various openness ratio datasets demonstrate that DCFS achieves state-of-art performance.

Figures (5)

• Figure 1: A possible case where softmax produces the same probabilities for known class "Cat" and unknown class "Lynx".
• Figure 2: Dirichlet distributions, data uncertainty, and distribution uncertainty of "Cat" and "Lynx" examples with different evidence-based Dirichlet parameters that are exponentially correlated to logits.
• Figure 3: The overall procedure of our proposed approach. In the model training phase, the discrepancy score $S^{dis}$ of the two classifier heads is amplified first with $\mathcal{L}_{un}$ and then we train the model with an evidential-based loss $\mathcal{L}_{edl}$. During the example selection stage, we select informative known-class examples with a combination of data uncertainty $U^{data}$, distribution uncertainty $U^{dist}$, and discrepancy score $S^{dis}$. Lastly, the selected $X^{query}$ is sent for oracle labeling.
• Figure 4: The curve of select accuracy of each cycle under CIFAR10 with 0.4 openness ratio.
• Figure 5: The t-NSE graph of selected known-class examples and existing labeled examples under CIFAR10 with 0.4 openness ratio.