Algorithmic Ways of Seeing: Using Object Detection to Facilitate Art Exploration

TL;DR

This work demonstrates that modern multimodal object-detection methods (GLIP/CLIP) can be applied to a large digital art collection to support open-ended exploration. The authors design SMKExplore, a web application that lets users browse paintings by detected objects, cluster objects by similarity, save favorites, and create new images via an outpainting canvas powered by DALL-E 2. Through on-site evaluation with 22 visitors, the study shows that object-centric exploration reveals recurring motifs, fosters attention to detail, and engages users in creative experimentation, while highlighting challenges around labelling bias and context. The findings offer design implications for label curation, trust management, and education, and point to future work extending beyond paintings and integrating more expert collaboration. The approach promises to broaden access and teach art appreciation by enabling visitors to discover artworks through the visual content of objects rather than solely through traditional metadata or keyword search.

Abstract

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Figures (9)

• Figure 1: Comparison of image analysis results.
• Figure 2: Comparison of image analysis results.
• Figure 3: Objects detected in "The King and Queen Surrounded by Swift Nudes" by Inge Ellegaard (1982).
• Figure 4: Screens from SMKExplore.
• Figure 5: Total number of visits per category throughout the 22 sessions.