Thin-Plate Spline-based Interpolation for Animation Line Inbetweening

TL;DR

The paper tackles animation line inbetweening for sparse line art with large inter-frame motion by introducing a raster-based pipeline that starts with thin-plate spline (TPS)–based coarse motion estimation from keypoint correspondences, producing a rough alignment between key frames via a mapping \(\mathcal{F}_{0\rightarrow 1}\) where \(F_{0\rightarrow 1}(p_1) = A p_1^T + \sum_i w_i \mathcal{U}(\|p_0^i - p_1\|_2)\) and \(\mathcal{U}(k) = k^2 \log k\). This coarse estimate is refined by a lightweight motion-refinement module using IFBlocks to compute residual flows and a fusion map, followed by multi-scale feature alignment and a UNet to synthesize the final inbetween frame \(\bm{y}_t\). To better reflect perceptual quality, the method introduces Weighted Chamfer Distance (WCD) and employs Earth Mover’s Distance (EMD) alongside a user study, showing superior performance over state-of-the-art methods on frame gaps \(N=1,5,9\). The approach is evaluated on the MixiamoLine240 dataset, with training at \(512\times512\) and testing at \(720\times720\), and the code is released at the authors’ repository. Overall, the TPS-based coarse motion estimation combined with a refinement stage yields more continuous and faithful line art interpolation, addressing the limitations of CD in line art contexts and providing a practical tool for animation pipelines.

Abstract

Animation line inbetweening is a crucial step in animation production aimed at enhancing animation fluidity by predicting intermediate line arts between two key frames. However, existing methods face challenges in effectively addressing sparse pixels and significant motion in line art key frames. In literature, Chamfer Distance (CD) is commonly adopted for evaluating inbetweening performance. Despite achieving favorable CD values, existing methods often generate interpolated frames with line disconnections, especially for scenarios involving large motion. Motivated by this observation, we propose a simple yet effective interpolation method for animation line inbetweening that adopts thin-plate spline-based transformation to estimate coarse motion more accurately by modeling the keypoint correspondence between two key frames, particularly for large motion scenarios. Building upon the coarse estimation, a motion refine module is employed to further enhance motion details before final frame interpolation using a simple UNet model. Furthermore, to more accurately assess the performance of animation line inbetweening, we refine the CD metric and introduce a novel metric termed Weighted Chamfer Distance, which demonstrates a higher consistency with visual perception quality. Additionally, we incorporate Earth Mover's Distance and conduct user study to provide a more comprehensive evaluation. Our method outperforms existing approaches by delivering high-quality interpolation results with enhanced fluidity. The code is available at \url{https://github.com/Tian-one/tps-inbetween}.

Figures (6)

• Figure 1: Common workflow of animation production, including creating key animation, inbetweening, colorizing and compositing.
• Figure 2: Visual comparison and corresponding metrics. The left side showcases the overlapped input frames along with the ground truth, while the right side displays the interpolation results and their evaluation metrics of inbetweening methods. TPS-forward and TPS-backward refer to the results obtained only using the flow estimated by the TPS module to warp the input's first and last frames, respectively. WCD is an enhanced metric that we proposed, which is better aligned with human visual perception, as detailed in Eq. \ref{['eq:WCD']}.
• Figure 3: A simple evaluation case of orange and black circular objects. In the presence of line disconnections, the CD metric is misleading.
• Figure 4: The overall pipeline of our approach begins with the initial estimation of coarse motion using a TPS-based transformation, which models point correspondence between key frames. Subsequently, we employ a simple motion refine module consists an optical flow network for estimating fine motion and a simple UNet model for synthesizing the inbetweening results.
• Figure 5: Inbetweening results comparison on the MixiamoLine240 dataset.