Accelerated Sub-Image Search For Variable-Size Patches Identification Based On Virtual Time Series Transformation And Segmentation

TL;DR

An acceleration mechanism for sub-image search that is based on a transformation of an image to multivariate time series along the RGB-channels and subsequent segmentation to reduce the 2D search space in the image is presented.

Abstract

This paper addresses two tasks: (i) fixed-size objects such as hay bales are to be identified in an aerial image for a given reference image of the object, and (ii) variable-size patches such as areas on fields requiring spot spraying or other handling are to be identified in an image for a given small-scale reference image. Both tasks are related. The second differs in that identified sub-images similar to the reference image are further clustered before patches contours are determined by solving a traveling salesman problem. Both tasks are complex in that the exact number of similar sub-images is not known a priori. The main discussion of this paper is presentation of an acceleration mechanism for sub-image search that is based on a transformation of an image to multivariate time series along the RGB-channels and subsequent segmentation to reduce the 2D search space in the image. Two variations of the acceleration mechanism are compared to exhaustive search on diverse synthetic and real-world images. Quantitatively, proposed method results in solve time reductions of up to 2 orders of magnitude, while qualitatively delivering comparative visual results. Proposed method is neural network-free and does not use any image pre-processing.

Figures (9)

• Figure 1: Problem 1: Graphical abstract. Fixed-size objects such as hay bales are to be identified in an image for a given smaller-scale reference image. Proposed hierarchical solution approach leverages a virtual time series transformation and segmentation to reduce the search space for sub-image search using similarity with respect to the reference image as filtering criterion. Note that the reduced search space shows the $(x,y)$-coordinates of the top-left corner of any sub-image measured in similarity to the reference image.
• Figure 2: Problem 2: Graphical abstract. Variable-size patches such as areas on fields requiring spraying or other handling are to be identified in an image for a given smaller-scale reference patch. The approach to address this problem is identical to the approach for Problem \ref{['problem1']}, however with the extension that identified sub-images are further clustered and contours are generated to produce independent non-overlapping patches areas.
• Figure 3: (a) Virtual time series and their segmentations along $x$- and $y$-channels, respectively. Each channel consists of 3 time series, one for each RGB-color. (b) Reduced search space using the segmentation instances plus a tolerance as sampling points for sub-image search. The reduced search space shows the $(x,y)$-coordinates of the top-left corner of any sub-image of dimension $N_x^\text{ref}\times N_y^\text{ref}$ measured in similarity to the reference image.
• Figure 4: Problem \ref{['problem2']}: After the identification of sub-images similar to the reference image these are further clustered before patches contours are determined by solving a traveling salesman problem.
• Figure 5: Experiments: Problem data. 10 images with small rectangular reference images inside are displayed. For Ex. 1 and 2 reference images are indicated by the transparent rectangles. For the remaining experiments reference image locations are indicated by white rectangles for better clarity. Reference images are enlarged in Fig. \ref{['fig_10refimg']}.