Multi Kernel Estimation based Object Segmentation

TL;DR

This paper introduces Multi-KernelGAN, which extends KernelGAN's capabilities by estimating two distinct kernels based on object segmentation masks, and demonstrates that this multi-kernel estimation technique outperforms conventional single-kernel methods in super-resolution tasks.

Abstract

This paper presents a novel approach for multi-kernel estimation by enhancing the KernelGAN algorithm, which traditionally estimates a single kernel for the entire image. We introduce Multi-KernelGAN, which extends KernelGAN's capabilities by estimating two distinct kernels based on object segmentation masks. Our approach is validated through three distinct methods: texture-based patch Fast Fourier Transform (FFT) calculation, detail-based segmentation, and deep learning-based object segmentation using YOLOv8 and the Segment Anything Model (SAM). Among these methods, the combination of YOLO and SAM yields the best results for kernel estimation. Experimental results demonstrate that our multi-kernel estimation technique outperforms conventional single-kernel methods in super-resolution tasks.

Figures (7)

• Figure 1: Multi-KernelGAN Model Pipeline.
• Figure 2: YOLOv8 and SAM-based Mask Creation. The YOLOv8 bounding boxes provide object localization, while SAM refines these regions with accurate segmentation.
• Figure 3: Frequency-Based Mask Using Fast Fourier Transform (FFT). The figure also shows the histogram of frequency values with a threshold: values to the left of the threshold map to Mask A (background), and values to the right map to Mask B (foreground).
• Figure 4: Details-Based Mask: From left to right, edges and contours mask, and creation of anchor pixels mask based on data-rich patches.
• Figure 6: Average performance differences between the Multi KernelGAN images and the ZSSR+KernelGAN. where positive values are for cases where Multi-KernelGAN outperformed KernelGAN.