Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond

TL;DR

The coupled thin-plate spline model (CoupledTPS), which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation, and a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data is developed.

Abstract

Thin-plate spline (TPS) is a principal warp that allows for representing elastic, nonlinear transformation with control point motions. With the increase of control points, the warp becomes increasingly flexible but usually encounters a bottleneck caused by undesired issues, e.g., content distortion. In this paper, we explore generic applications of TPS in single-image-based warping tasks, such as rotation correction, rectangling, and portrait correction. To break this bottleneck, we propose the coupled thin-plate spline model (CoupledTPS), which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. Concretely, we first design an iterative search to predict new control points according to the current latent condition. Then, we present the warping flow as a bridge for the coupling of different TPS transformations, effectively eliminating interpolation errors caused by multiple warps. Besides, in light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. It is formulated through dual transformation between the searched control points of unlabeled data and its graphic augmentation, yielding an implicit correction consistency constraint. Finally, we collect massive unlabeled data to exhibit the benefit of our semi-supervised scheme in rotation correction. Extensive experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art (SoTA) solutions for rotation correction and beyond. The code and data are available at https://github.com/nie-lang/CoupledTPS.

Figures (12)

• Figure 1: Three examples of our method. The proposed CoupledTPS corrects the 2D in-plane tilt, irregular boundaries, and wide-angle portrait via a unified warping framework.
• Figure 2: The visualization of the performance bottleneck in vanilla TPS. With the increase of the control point number, the corrected results are prone to produce content distortions.
• Figure 3: The workflow of the proposed CoupledTPS. It first encodes an input image into the latent condition and then predicts the source control points from the latent condition. The predicted warp for each iteration is used to update the latent condition for the next iteration. The warping flow is leveraged to couple the currently predicted warp with the previously coupled warp and eliminate interpolation errors. The initial flow is set to 0.
• Figure 4: Comparison of control point prediction strategies. (a) Predicting massive control points at once. (b) Predicting residual motions iteratively like RAFT teed2020raft. (c) Predicting different control points iteratively (ours).
• Figure 5: Dual transformation for unlabeled data. We design the dual transformation between the unlabeled data and its graphic augmentation to establish the implicit correction consistency constraint.