Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Object Remover Performance Evaluation Methods using Class-wise Object Removal Images

Changsuk Oh, Dongseok Shim, Taekbeom Lee, H. Jin Kim

TL;DR

The paper tackles the challenge of evaluating object remover quality, arguing that traditional full-reference metrics using original images are ill-suited for object removal tasks. It introduces a class-wise, unpaired evaluation framework with $FID^*$ and $U-IDS^*$ that rely on a comparison set of images without the target class, avoiding the need for object removal ground truth. Through experiments on COCO and a self-acquired CARLA dataset, the proposed metrics align with human judgments and GT-based evaluations, addressing biases seen in conventional metrics. This approach enables robust model selection and performance assessment for object removers across diverse real and synthetic image styles without requiring GT for every instance.

Abstract

Object removal refers to the process of erasing designated objects from an image while preserving the overall appearance, and it is one area where image inpainting is widely used in real-world applications. The performance of an object remover is quantitatively evaluated by measuring the quality of object removal results, similar to how the performance of an image inpainter is gauged. Current works reporting quantitative performance evaluations utilize original images as references. In this letter, to validate the current evaluation methods cannot properly evaluate the performance of an object remover, we create a dataset with object removal ground truth and compare the evaluations made by the current methods using original images to those utilizing object removal ground truth images. The disparities between two evaluation sets validate that the current methods are not suitable for measuring the performance of an object remover. Additionally, we propose new evaluation methods tailored to gauge the performance of an object remover. The proposed methods evaluate the performance through class-wise object removal results and utilize images without the target class objects as a comparison set. We confirm that the proposed methods can make judgments consistent with human evaluators in the COCO dataset, and that they can produce measurements aligning with those using object removal ground truth in the self-acquired dataset.

Object Remover Performance Evaluation Methods using Class-wise Object Removal Images

TL;DR

The paper tackles the challenge of evaluating object remover quality, arguing that traditional full-reference metrics using original images are ill-suited for object removal tasks. It introduces a class-wise, unpaired evaluation framework with and that rely on a comparison set of images without the target class, avoiding the need for object removal ground truth. Through experiments on COCO and a self-acquired CARLA dataset, the proposed metrics align with human judgments and GT-based evaluations, addressing biases seen in conventional metrics. This approach enables robust model selection and performance assessment for object removers across diverse real and synthetic image styles without requiring GT for every instance.

Abstract

Object removal refers to the process of erasing designated objects from an image while preserving the overall appearance, and it is one area where image inpainting is widely used in real-world applications. The performance of an object remover is quantitatively evaluated by measuring the quality of object removal results, similar to how the performance of an image inpainter is gauged. Current works reporting quantitative performance evaluations utilize original images as references. In this letter, to validate the current evaluation methods cannot properly evaluate the performance of an object remover, we create a dataset with object removal ground truth and compare the evaluations made by the current methods using original images to those utilizing object removal ground truth images. The disparities between two evaluation sets validate that the current methods are not suitable for measuring the performance of an object remover. Additionally, we propose new evaluation methods tailored to gauge the performance of an object remover. The proposed methods evaluate the performance through class-wise object removal results and utilize images without the target class objects as a comparison set. We confirm that the proposed methods can make judgments consistent with human evaluators in the COCO dataset, and that they can produce measurements aligning with those using object removal ground truth in the self-acquired dataset.
Paper Structure (6 sections, 2 equations, 5 figures, 1 table)

This paper contains 6 sections, 2 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Object Remover Performance Evaluation
  3. Current object remover performance evaluation methods
  4. Proposed method
  5. Experiments
  6. Conclusion

Figures (5)

  • Figure 1: Object removal results on the COCO dataset. We use Lama suvorov2022resolution for removal and human class objects are removed. Previous evaluation methods judge that the object remover using $kernel\_size=0$ performs the best, while our methods evaluate that the object remover using $kernel\_size=10$ erased the removal target in the most plausible way.
  • Figure 2: Object remover performance evaluations on the proposed CARLA dataset. We use Lama suvorov2022resolution for removal. The score of each method is divided by the highest score of each method. $\uparrow$ and $\downarrow$ indicate higher is better and lower is better, respectively. The object remover which is judged to have the best performance for each evaluation method is indicated with a rectangle.
  • Figure 3: Image samples of the CARLA dataset with object removal ground truth.
  • Figure 4: Performance evaluations made by the unpaired data methods (FID and U-IDS) and proposed methods (FID$^*$ and U-IDS$^*$) on the COCO dataset. The score of each method is divided by the highest score of each method. $\uparrow$ and $\downarrow$ indicate higher is better and lower is better, respectively. The object remover which is judged to have the best performance for each evaluation method is indicated with a rectangle.
  • Figure 5: RSD values of FID$^*$ and U-IDS$^*$ obtained by setting the number of random samples differently. Results are obtained through 20 iterations. Lama suvorov2022resolution, CR-Fill zeng2021cr, MAT li2022mat, MADF zhu2021image, and RePaint lugmayr2022repaint are utilized to generate object removal results.