Enhanced-alignment Measure for Binary Foreground Map Evaluation

TL;DR

This paper takes a detailed look at current binary FM evaluation measures and proposes a novel and effective E-measure (Enhanced-alignment measure), jointly capturing image-level statistics and local pixel matching information, which demonstrates the superiority of this measure over the available measures on 4 popular datasets via 5 meta-measures.

Abstract

The existing binary foreground map (FM) measures to address various types of errors in either pixel-wise or structural ways. These measures consider pixel-level match or image-level information independently, while cognitive vision studies have shown that human vision is highly sensitive to both global information and local details in scenes. In this paper, we take a detailed look at current binary FM evaluation measures and propose a novel and effective E-measure (Enhanced-alignment measure). Our measure combines local pixel values with the image-level mean value in one term, jointly capturing image-level statistics and local pixel matching information. We demonstrate the superiority of our measure over the available measures on 4 popular datasets via 5 meta-measures, including ranking models for applications, demoting generic, random Gaussian noise maps, ground-truth switch, as well as human judgments. We find large improvements in almost all the meta-measures. For instance, in terms of application ranking, we observe improvementrangingfrom9.08% to 19.65% compared with other popular measures.

Figures (13)

• Figure 1: Inaccuracy of current evaluation measures. A measure should score the FM (c) generated by a state-of-the-art algorithm a higher value than the random Gaussian noise map (d). Current common measures including IOU [Everingham et al., 2010], F1/JI [Jaccard, 1901] , Fbw [Margolin et al., 2014], CM [Movahedi and Elder, 2010], and VQ [Shi et al., 2015] prefer the noise map. Only our measure correctly ranked (c) higher than (d).
• Figure 2: Demonstration of effectiveness of our measure. The ranking of binary foreground maps (after threshold) are generated by 3 state-of-the-art salient object detection models including DCL [Li and Yu, 2016], RFCN [Wang et al., 2016] and DHS [Liu and Han, 2016]. All of the 3 different types of popular measures (CM, Fbw and VQ) fail to rank the maps correctly. However, our measure gives the right order.
• Figure 3: Limitations of region-based measures. The blue circle represents GT and red curve denotes FM. Based on IOU, F1/JI measures, the intersection in (b) is almost equal to the intersection in (a) when compared with GT circle (blue circle curve), although it has spikes, wiggles and shape differences [Movahedi and Elder, 2010].
• Figure 4: Our E-measure framework. (a) ground-truth map. (b) the estimate foreground map. (c) & (d) are the mean values map of GT & FM, respectively. (e) and (f) are the bias matrices calculated by (Eq. 4). (g) is the mapping function. (h) is the enhanced alignment matrix computed by (Eq. 6). 'aligned' & 'unaligned' donate those points which$G T(x, y)=F M(x, y) \& G T(x, y) \neq F M(x, y)$, respectively.
• Figure 5: Application Ranking. To rank foreground maps according to an application, we compare the output when using the GT, to the output when using the FM foreground map. The more similar an FM foreground map is to a GT map, the closer its application's output should be to GT output.