Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The ArtBench Dataset: Benchmarking Generative Models with Artworks

Peiyuan Liao, Xiuyu Li, Xihui Liu, Kurt Keutzer

TL;DR

ArtBench-10 addresses a gap in artwork generation benchmarks by providing a standardized, class-balanced dataset of 60,000 artworks across 10 styles, with 3 target resolutions and a uniform preprocessing pipeline. It details an end-to-end data collection, annotation, and sampling pipeline to produce balanced, high-quality training and testing splits, and benchmarks a range of generative models (GANs, diffusion, VAEs) using IS, FID, KID, and improved precision/recall metrics. The results show strong performance for StyleGAN2-ADA across settings, while revealing trade-offs between quality and diversity across models and styles, and confirming non-memorization via nearest-neighbor analysis. The dataset aims to standardize artwork synthesis evaluation, while acknowledging biases toward European, North American, and East Asian art and outlining plans for broader coverage and responsible use in future work.

Abstract

We introduce ArtBench-10, the first class-balanced, high-quality, cleanly annotated, and standardized dataset for benchmarking artwork generation. It comprises 60,000 images of artwork from 10 distinctive artistic styles, with 5,000 training images and 1,000 testing images per style. ArtBench-10 has several advantages over previous artwork datasets. Firstly, it is class-balanced while most previous artwork datasets suffer from the long tail class distributions. Secondly, the images are of high quality with clean annotations. Thirdly, ArtBench-10 is created with standardized data collection, annotation, filtering, and preprocessing procedures. We provide three versions of the dataset with different resolutions ($32\times32$, $256\times256$, and original image size), formatted in a way that is easy to be incorporated by popular machine learning frameworks. We also conduct extensive benchmarking experiments using representative image synthesis models with ArtBench-10 and present in-depth analysis. The dataset is available at https://github.com/liaopeiyuan/artbench under a Fair Use license.

The ArtBench Dataset: Benchmarking Generative Models with Artworks

TL;DR

ArtBench-10 addresses a gap in artwork generation benchmarks by providing a standardized, class-balanced dataset of 60,000 artworks across 10 styles, with 3 target resolutions and a uniform preprocessing pipeline. It details an end-to-end data collection, annotation, and sampling pipeline to produce balanced, high-quality training and testing splits, and benchmarks a range of generative models (GANs, diffusion, VAEs) using IS, FID, KID, and improved precision/recall metrics. The results show strong performance for StyleGAN2-ADA across settings, while revealing trade-offs between quality and diversity across models and styles, and confirming non-memorization via nearest-neighbor analysis. The dataset aims to standardize artwork synthesis evaluation, while acknowledging biases toward European, North American, and East Asian art and outlining plans for broader coverage and responsible use in future work.

Abstract

We introduce ArtBench-10, the first class-balanced, high-quality, cleanly annotated, and standardized dataset for benchmarking artwork generation. It comprises 60,000 images of artwork from 10 distinctive artistic styles, with 5,000 training images and 1,000 testing images per style. ArtBench-10 has several advantages over previous artwork datasets. Firstly, it is class-balanced while most previous artwork datasets suffer from the long tail class distributions. Secondly, the images are of high quality with clean annotations. Thirdly, ArtBench-10 is created with standardized data collection, annotation, filtering, and preprocessing procedures. We provide three versions of the dataset with different resolutions (, , and original image size), formatted in a way that is easy to be incorporated by popular machine learning frameworks. We also conduct extensive benchmarking experiments using representative image synthesis models with ArtBench-10 and present in-depth analysis. The dataset is available at https://github.com/liaopeiyuan/artbench under a Fair Use license.
Paper Structure (84 sections, 12 figures, 6 tables, 1 algorithm)

This paper contains 84 sections, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Synthesis
  4. Image Synthesis Datasets and Benchmarks
  5. Artworks Datasets and Benchmarks
  6. The ArtBench-10 Dataset
  7. Limitation of Existing Artwork Datasets
  8. Dataset Creation
  9. Data collection
  10. Data annotation and filtering
  11. Filtering out near-duplicates and low-quality images
  12. Balanced Sampling
  13. Standardized formatting.
  14. Dataset Statistics and Analysis
  15. Experiments
  16. ...and 69 more sections

Figures (12)

  • Figure 1: Overview of the 10 artistic styles and corresponding images in ArtBench-10. ArtBench-10 is a class-balanced dataset with 6,000 images for each of the 10 artistic styles.
  • Figure 2: Issues with the scraped WikiArt data (97908 images as in April 2022). (Left) Long tail distribution over classes. (Right) A pair of duplicate images in the dataset.
  • Figure 3: ArtBench data collection process.
  • Figure 4: Statistics for ArtBench-10. Our dataset covers a wide of artworks across different time periods and greatly reduces imbalanced among artists for diversity.
  • Figure 5: (Left) Confusion matrix on the test set with SEResNet-34 trained on ArtBench-10 ($256 \times 256$); (Right) Distribution of aspect ratios from raw data (top) and ArtBench-10 (bottom), respectively.
  • ...and 7 more figures