Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

APISR: Anime Production Inspired Real-World Anime Super-Resolution

Boyang Wang, Fengyu Yang, Xihang Yu, Chao Zhang, Hanbin Zhao

TL;DR

Problem: Real-world anime SR suffers when photorealistic SR methods are naively applied to frame-based animation. Approach: APISR uses an image-centric pipeline with API dataset, a prediction-oriented degradation model, hand-drawn line enhancement, and a balanced twin perceptual loss. Contributions: API dataset creation (3,740 images from I-Frames with ICA-based selection and 720P back-original), a degradation model simulating video artifacts from single images, pseudo-GT line enhancement via XDoG, and a loss that blends ResNet50 anime features with photorealistic VGG features. Findings: APISR outperforms SOTA on AVC-RealLQ with a compact model and improved restoration quality, reducing training data needs.

Abstract

While real-world anime super-resolution (SR) has gained increasing attention in the SR community, existing methods still adopt techniques from the photorealistic domain. In this paper, we analyze the anime production workflow and rethink how to use characteristics of it for the sake of the real-world anime SR. First, we argue that video networks and datasets are not necessary for anime SR due to the repetition use of hand-drawing frames. Instead, we propose an anime image collection pipeline by choosing the least compressed and the most informative frames from the video sources. Based on this pipeline, we introduce the Anime Production-oriented Image (API) dataset. In addition, we identify two anime-specific challenges of distorted and faint hand-drawn lines and unwanted color artifacts. We address the first issue by introducing a prediction-oriented compression module in the image degradation model and a pseudo-ground truth preparation with enhanced hand-drawn lines. In addition, we introduce the balanced twin perceptual loss combining both anime and photorealistic high-level features to mitigate unwanted color artifacts and increase visual clarity. We evaluate our method through extensive experiments on the public benchmark, showing our method outperforms state-of-the-art anime dataset-trained approaches.

APISR: Anime Production Inspired Real-World Anime Super-Resolution

TL;DR

Problem: Real-world anime SR suffers when photorealistic SR methods are naively applied to frame-based animation. Approach: APISR uses an image-centric pipeline with API dataset, a prediction-oriented degradation model, hand-drawn line enhancement, and a balanced twin perceptual loss. Contributions: API dataset creation (3,740 images from I-Frames with ICA-based selection and 720P back-original), a degradation model simulating video artifacts from single images, pseudo-GT line enhancement via XDoG, and a loss that blends ResNet50 anime features with photorealistic VGG features. Findings: APISR outperforms SOTA on AVC-RealLQ with a compact model and improved restoration quality, reducing training data needs.

Abstract

While real-world anime super-resolution (SR) has gained increasing attention in the SR community, existing methods still adopt techniques from the photorealistic domain. In this paper, we analyze the anime production workflow and rethink how to use characteristics of it for the sake of the real-world anime SR. First, we argue that video networks and datasets are not necessary for anime SR due to the repetition use of hand-drawing frames. Instead, we propose an anime image collection pipeline by choosing the least compressed and the most informative frames from the video sources. Based on this pipeline, we introduce the Anime Production-oriented Image (API) dataset. In addition, we identify two anime-specific challenges of distorted and faint hand-drawn lines and unwanted color artifacts. We address the first issue by introducing a prediction-oriented compression module in the image degradation model and a pseudo-ground truth preparation with enhanced hand-drawn lines. In addition, we introduce the balanced twin perceptual loss combining both anime and photorealistic high-level features to mitigate unwanted color artifacts and increase visual clarity. We evaluate our method through extensive experiments on the public benchmark, showing our method outperforms state-of-the-art anime dataset-trained approaches.
Paper Structure (28 sections, 3 equations, 18 figures, 4 tables)

This paper contains 28 sections, 3 equations, 18 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Realated Work
  3. Real-World Super-Resolution.
  4. Anime Processing.
  5. Proposed Method
  6. Anime Production-Oriented Image SR Dataset
  7. An Anime Practical Degradation model
  8. Anime Hand-Drawn Lines Enhancement
  9. Balanced Twin Perceptual Loss for Anime
  10. Experiment
  11. Implementation Details
  12. Comparisons with State-of-the-art Methods
  13. Quantitative Comparison.
  14. Qualitative Comparison.
  15. Ablation Study
  16. ...and 13 more sections

Figures (18)

  • Figure 1: Comparisons between proposed APISR and other SOTA anime SR methods. Ours present clearer and sharper hand-drawn lines, better restoration with more natural details, and do not present unwanted color artifacts. Zoom in for best view.
  • Figure 2: We identify two new anime-specific challenges: (a) Distorted and faint hand-drawn lines frequently appear in real-world anime images. (b) Unwanted color artifacts in AnimeSR wu2022animesr and VQD-SR tuo2023learning. Zoom in for the best view.
  • Figure 3: Histogram of (a) the average image data size comparison between I-Frames and Non-I-Frames (P and B-Frame) in collected video sources and (b) image complexity feng2022ic9600 comparison between proposed API and AVC wu2022animesr dataset.
  • Figure 4: Image Quality Assessment (IQA) with HyperIQA su2020blindly and Brisque mittal2012no vs. Image Complexity Assessment (ICA) with IC9600 feng2022ic9600. IQA favors simple scenes and gives low scores to images with strong CGI. However, ICA is the opposite.
  • Figure 5: Samples of API Super-Resolution Dataset. API includes versatile CGI effects scenes (e.g., different lightning and special effects) and presents high image complexity.
  • ...and 13 more figures