Evaluation in Neural Style Transfer: A Review

TL;DR

This paper tackles the challenge of evaluating Neural Style Transfer (NST) by surveying the diverse qualitative, human, and quantitative evaluation methods across image and video NST, and by highlighting the lack of standardized benchmarks. It analyzes datasets, evaluation designs, and metrics, exposing inconsistencies that hinder fair comparisons and reproducibility. The authors propose concrete recommendations, including benchmark datasets for content/style and evaluation-specific data, standardized human studies, and a concise set of quantitative metrics, aimed at enabling reliable, interpretable, and repeatable NST assessments. The work emphasizes open data practices and statistical rigor as essential steps toward a unified evaluation framework, which could significantly accelerate fair progress in NST research and applications.

Abstract

The field of Neural Style Transfer (NST) has witnessed remarkable progress in the past few years, with approaches being able to synthesize artistic and photorealistic images and videos of exceptional quality. To evaluate such results, a diverse landscape of evaluation methods and metrics is used, including authors' opinions based on side-by-side comparisons, human evaluation studies that quantify the subjective judgements of participants, and a multitude of quantitative computational metrics which objectively assess the different aspects of an algorithm's performance. However, there is no consensus regarding the most suitable and effective evaluation procedure that can guarantee the reliability of the results. In this review, we provide an in-depth analysis of existing evaluation techniques, identify the inconsistencies and limitations of current evaluation methods, and give recommendations for standardized evaluation practices. We believe that the development of a robust evaluation framework will not only enable more meaningful and fairer comparisons among NST methods but will also enhance the comprehension and interpretation of research findings in the field.

Figures (8)

• Figure 1: Evaluation in NST: Qualitative Evaluation and Quantitative Evaluation. To align with current literature, we use three main categories for evaluation, highlighted in blue.
• Figure 2: The utilization of Quantitative Computational Metrics and User Studies by the NST methods. The graph shows the number of Image and Video NST methods that employed each of the quantitative metrics to evaluate their results and the amount of methods that conducted User Studies with Expert and Non-Expert participants. The graph shows that each method employs only a small selection of quantitative metrics for evaluative comparisons; some metrics are utilized more than others. At the time of this survey, the recently proposed ArtScorechen2023learning metric has not been utilized by any of the reviewed approaches.
• Figure 3: Evaluation Techniques utilized in the NST literature categorized into Qualitative Evaluation, Human Evaluation Studies, and Quantitative Metrics. The blue highlighting reflects the standard categorization in the literature (Figure \ref{['fig:evaluation-general-categories']}).
• Figure 4: The variation in methods comparing against, number of participants recruited, and response format in User Studies of Artistic Image NST approaches. "N/A" in the second graph denotes that a study has not provided the relevant information.
• Figure 5: The number of Image and Video NST studies including Ablation Studies. The graph depicts the reviewed papers of the NST literature.