PairingNet: A Learning-based Pair-searching and -matching Network for Image Fragments

TL;DR

PairingNet addresses the challenging problem of searching and matching pairs among large collections of irregular image fragments for restoration tasks. It combines a graph-based local feature extractor (contour and texture) with a weighted fusion scheme and a linear-transformer based global search to achieve efficient, accurate pair-searching and pair-matching. The approach is validated on a generated irregular-fragment dataset and a real-fragment dataset, showing superior performance and significantly faster inference than baselines. The work provides extensive ablations and visual analyses, along with a dataset generation algorithm to support data-driven learning for fragment restoration with potential extensions to related reconstruction tasks.

Abstract

In this paper, we propose a learning-based image fragment pair-searching and -matching approach to solve the challenging restoration problem. Existing works use rule-based methods to match similar contour shapes or textures, which are always difficult to tune hyperparameters for extensive data and computationally time-consuming. Therefore, we propose a neural network that can effectively utilize neighbor textures with contour shape information to fundamentally improve performance. First, we employ a graph-based network to extract the local contour and texture features of fragments. Then, for the pair-searching task, we adopt a linear transformer-based module to integrate these local features and use contrastive loss to encode the global features of each fragment. For the pair-matching task, we design a weighted fusion module to dynamically fuse extracted local contour and texture features, and formulate a similarity matrix for each pair of fragments to calculate the matching score and infer the adjacent segment of contours. To faithfully evaluate our proposed network, we created a new image fragment dataset through an algorithm we designed that tears complete images into irregular fragments. The experimental results show that our proposed network achieves excellent pair-searching accuracy, reduces matching errors, and significantly reduces computational time. Details, sourcecode, and data are available in our supplementary material.

Figures (19)

• Figure 1: Our pair-searching and -matching task. Given a large number of mixed image fragments, our proposed network can search and match pairs of fragments, which is important for restoration problems. We also designed an algorithm to generate a dataset of image fragments by tearing a set of complete images.
• Figure 1: Cutting examples in our generated dataset, which are rich in shapes and patterns.
• Figure 2: Pipeline of our proposed PairingNet. (a) Based on patches, we first employ a binary encoding to describe contours and use ResGCN as the backbone to extract the local contour and texture features of image fragments. (b) We design self-gated fusion to fuse the extracted features in a weighted manner to calculate the similarity matrix for each pair of fragments, and (c) a linear transformer-based encoder for learning the global features of each fragment. (d) We use a two-step strategy to train our proposed network. (e) During inference, we calculate cosine similarity to find the adjacent fragment pairs and further process the matching similarity matrix to establish better correspondences between a fragment pair.
• Figure 2: (a) The process of generating real dataset. (b) More examples of processing a complete real image into multiple fragments.
• Figure 3: Dataset creation. (a) The process of generating our dataset. (b) Illustration of Algirithm \ref{['alg:data generate']} for tearing a complete image.