Coarse-to-Fine Non-rigid Multi-modal Image Registration for Historical Panel Paintings based on Crack Structures

TL;DR

The paper addresses automatic multi-modal registration of historical panel paintings, a task challenged by extremely large, variably-resolved images and non-rigid distortions. It introduces a coarse-to-fine framework that leverages craquelure-based keypoints from CraquelureNet, sub-pixel refinement via Craquelure Refine, and LightGlue for robust sparse matching, followed by TPS-based non-rigid warping. A two-stage refinement strategy further scales keypoints to higher resolutions through score-map and feature-volume correlations, aided by dedicated training regimes. Extensive experiments on a newly curated multi-modal panel painting dataset show state-of-the-art accuracy and efficiency, with robust performance across VIS, IRR, UV, XR, and MACRO modalities and substantial improvements over dense matching baselines.

Abstract

Art technological investigations of historical panel paintings rely on acquiring multi-modal image data, including visual light photography, infrared reflectography, ultraviolet fluorescence photography, x-radiography, and macro photography. For a comprehensive analysis, the multi-modal images require pixel-wise alignment, which is still often performed manually. Multi-modal image registration can reduce this laborious manual work, is substantially faster, and enables higher precision. Due to varying image resolutions, huge image sizes, non-rigid distortions, and modality-dependent image content, registration is challenging. Therefore, we propose a coarse-to-fine non-rigid multi-modal registration method efficiently relying on sparse keypoints and thin-plate-splines. Historical paintings exhibit a fine crack pattern, called craquelure, on the paint layer, which is captured by all image systems and is well-suited as a feature for registration. In our one-stage non-rigid registration approach, we employ a convolutional neural network for joint keypoint detection and description based on the craquelure and a graph neural network for descriptor matching in a patch-based manner, and filter matches based on homography reprojection errors in local areas. For coarse-to-fine registration, we introduce a novel multi-level keypoint refinement approach to register mixed-resolution images up to the highest resolution. We created a multi-modal dataset of panel paintings with a high number of keypoint annotations, and a large test set comprising five multi-modal domains and varying image resolutions. The ablation study demonstrates the effectiveness of all modules of our refinement method. Our proposed approaches achieve the best registration results compared to competing keypoint and dense matching methods and refinement methods.

Figures (12)

• Figure 1: Our one-stage multi-modal non-rigid registration pipeline works on patches to detect good correspondences in local areas based on homography reprojection errors. Crack-based keypoints and descriptors are extracted using CraquelureNet and are refined to sub-pixel using our Craquelure Refine module. LightGlue is applied for matching and weighted DLT for homography estimation. All inlier point pairs are collected for all patches, are filtered using VFC, and a global homography is computed. The displacement vectors between the warped source and target points are used to compute a TPS.
• Figure 2: CraquelureNet extracts keypoints and descriptors based on the crack structure in multi-modal images of historic paintings. It consists of a ResNet backbone and a detection and description head. The keypoint heatmap is bicubically upscaled and post-processed using non-maximum suppression (NMS). The descriptors are bilinearly interpolated at the keypoint locations in the L2-normalized descriptor map.
• Figure 3: The Craquelure Refine module extends the ResNet backbone of CraquelureNet with the Detection refine head to a U-Net for sub-pixel keypoint refinement.
• Figure 4: Feature fusion head to extract craquelure features for normalized cross-correlation (NCC) refinement and for our adapted correlation-based LoFTR fine matching module.
• Figure 5: Success rates of ME and MAE for VIS-XR and XR-IRR registration with error threshold $\epsilon$: (a,b) VIS-XR Set1 (s1), (c,d) VIS-XR Set2 (s2), (e,f) VIS-XR Set3 (s2), (g,h) XR-IRR (s2). VFC is applied as post-processing. Methods with '-P' are fine-tuned on our Multi-modal Panel Painting dataset.