Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Iconographic Classification and Content-Based Recommendation for Digitized Artworks

Krzysztof Kutt, Maciej Baczyński

TL;DR

The paper tackles the need for interpretable iconographic access in digitized cultural heritage by introducing CARIS, a four-stage pipeline that unifies visual content analysis with Iconclass semantics. It combines YOLOv8 object detection, two mapping strategies to Iconclass codes, a lightweight rule-based inference for abstract meanings, and three content-based recommenders (hierarchy-based, IDF-weighted overlap, and Jaccard) to propose related artworks. Key contributions include a robust mapping workflow, a transparent rule engine, and a trio of semantic-aware recommenders evaluated on public Iconclass resources and exemplar artworks, highlighting both strengths and limitations of current CV and symbolic approaches. The system demonstrates potential to accelerate cataloging and improve navigation in large heritage collections, while outlining future directions in training data, multimodal integration, explainability, and interface design to help curators and users understand recommendations.

Abstract

We present a proof-of-concept system that automates iconographic classification and content-based recommendation of digitized artworks using the Iconclass vocabulary and selected artificial intelligence methods. The prototype implements a four-stage workflow for classification and recommendation, which integrates YOLOv8 object detection with algorithmic mappings to Iconclass codes, rule-based inference for abstract meanings, and three complementary recommenders (hierarchical proximity, IDF-weighted overlap, and Jaccard similarity). Although more engineering is still needed, the evaluation demonstrates the potential of this solution: Iconclass-aware computer vision and recommendation methods can accelerate cataloging and enhance navigation in large heritage repositories. The key insight is to let computer vision propose visible elements and to use symbolic structures (Iconclass hierarchy) to reach meaning.

Iconographic Classification and Content-Based Recommendation for Digitized Artworks

TL;DR

The paper tackles the need for interpretable iconographic access in digitized cultural heritage by introducing CARIS, a four-stage pipeline that unifies visual content analysis with Iconclass semantics. It combines YOLOv8 object detection, two mapping strategies to Iconclass codes, a lightweight rule-based inference for abstract meanings, and three content-based recommenders (hierarchy-based, IDF-weighted overlap, and Jaccard) to propose related artworks. Key contributions include a robust mapping workflow, a transparent rule engine, and a trio of semantic-aware recommenders evaluated on public Iconclass resources and exemplar artworks, highlighting both strengths and limitations of current CV and symbolic approaches. The system demonstrates potential to accelerate cataloging and improve navigation in large heritage collections, while outlining future directions in training data, multimodal integration, explainability, and interface design to help curators and users understand recommendations.

Abstract

We present a proof-of-concept system that automates iconographic classification and content-based recommendation of digitized artworks using the Iconclass vocabulary and selected artificial intelligence methods. The prototype implements a four-stage workflow for classification and recommendation, which integrates YOLOv8 object detection with algorithmic mappings to Iconclass codes, rule-based inference for abstract meanings, and three complementary recommenders (hierarchical proximity, IDF-weighted overlap, and Jaccard similarity). Although more engineering is still needed, the evaluation demonstrates the potential of this solution: Iconclass-aware computer vision and recommendation methods can accelerate cataloging and enhance navigation in large heritage repositories. The key insight is to let computer vision propose visible elements and to use symbolic structures (Iconclass hierarchy) to reach meaning.
Paper Structure (16 sections, 2 equations, 5 figures)

This paper contains 16 sections, 2 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related works
  3. Iconography and Iconclass
  4. Computer vision methods in CH pipelines
  5. Automatic iconographic classification
  6. Artwork recommendation
  7. IDF and Jaccard similarity
  8. System overview
  9. Classification
  10. Recommendation
  11. Evaluation
  12. Classification
  13. Recommendation
  14. Summary and future work
  15. Acknowledgments.
  16. ...and 1 more sections

Figures (5)

  • Figure 1: "The Aldrovandi Dog" by Guercino. Wikimedia Commons, public domain.
  • Figure 2: "The Hunting With Falcon" (Polish: "Wyjazd na polowanie z sokołem") by Juliusz Kossak. Wikimedia Commons, public domain.
  • Figure 3: Recommendations for "The Hunting With Falcon" based on detected codes (horse, human; classification missed falcon and dog). Both hierarchy-based similarity and IDF-weighted overlap returned IIHIM_-467547872.jpg file (left). Jaccard similarity returned IIHIM_RIJKS_-2107924074.jpg (right).
  • Figure 4: Recommendations for "The Hunting With Falcon" based on manual codes: dog, horse (+ man and animal), hoofed animals. Both hierarchy-based similarity and IDF-weighted overlap returned IIHIM_1579845581.jpg file (left). Jaccard similarity returned IIHIM_891269882.jpg (right).
  • Figure 5: Recommendations for "94L53 Hercules discovers Tiryns' famous dye: the muzzle of Hercules' dog is stained with purple after it has bitten into a mollusc" code. Only hierarchy-based similarity method was able to recommend an image from the Iconclass AI Test Set: embepu_f1691_pic0304.jpg.