Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jewelry Recognition via Encoder-Decoder Models

José M. Alcalde-Llergo, Enrique Yeguas-Bolívar, Andrea Zingoni, Alejandro Fuerte-Jurado

TL;DR

This work tackles jewelry recognition by reframing it as an image-captioning task using encoder–decoder models to generate detailed natural-language descriptions of jewelry, which are then used for classification at varying levels of detail. The authors build a Córdoba-based jewelry image dataset and systematically evaluate multiple CNN backbones and RNN decoders, finding that VGG-16 paired with GRU generally yields strong performance for both captioning and classification; the best captioning model attains about 0.957 test CCR. The study demonstrates practical potential for e-commerce and inventory-management applications, including a web interface for generating basic, normal, and complete descriptions, though earrings remain the most challenging category. Overall, the approach provides a pathway to automated, expert-like jewelry descriptions that can enhance user-facing applications and automated analysis of consumer tastes.

Abstract

Jewelry recognition is a complex task due to the different styles and designs of accessories. Precise descriptions of the various accessories is something that today can only be achieved by experts in the field of jewelry. In this work, we propose an approach for jewelry recognition using computer vision techniques and image captioning, trying to simulate this expert human behavior of analyzing accessories. The proposed methodology consist on using different image captioning models to detect the jewels from an image and generate a natural language description of the accessory. Then, this description is also utilized to classify the accessories at different levels of detail. The generated caption includes details such as the type of jewel, color, material, and design. To demonstrate the effectiveness of the proposed method in accurately recognizing different types of jewels, a dataset consisting of images of accessories belonging to jewelry stores in Córdoba (Spain) has been created. After testing the different image captioning architectures designed, the final model achieves a captioning accuracy of 95\%. The proposed methodology has the potential to be used in various applications such as jewelry e-commerce, inventory management or automatic jewels recognition to analyze people's tastes and social status.

Jewelry Recognition via Encoder-Decoder Models

TL;DR

This work tackles jewelry recognition by reframing it as an image-captioning task using encoder–decoder models to generate detailed natural-language descriptions of jewelry, which are then used for classification at varying levels of detail. The authors build a Córdoba-based jewelry image dataset and systematically evaluate multiple CNN backbones and RNN decoders, finding that VGG-16 paired with GRU generally yields strong performance for both captioning and classification; the best captioning model attains about 0.957 test CCR. The study demonstrates practical potential for e-commerce and inventory-management applications, including a web interface for generating basic, normal, and complete descriptions, though earrings remain the most challenging category. Overall, the approach provides a pathway to automated, expert-like jewelry descriptions that can enhance user-facing applications and automated analysis of consumer tastes.

Abstract

Jewelry recognition is a complex task due to the different styles and designs of accessories. Precise descriptions of the various accessories is something that today can only be achieved by experts in the field of jewelry. In this work, we propose an approach for jewelry recognition using computer vision techniques and image captioning, trying to simulate this expert human behavior of analyzing accessories. The proposed methodology consist on using different image captioning models to detect the jewels from an image and generate a natural language description of the accessory. Then, this description is also utilized to classify the accessories at different levels of detail. The generated caption includes details such as the type of jewel, color, material, and design. To demonstrate the effectiveness of the proposed method in accurately recognizing different types of jewels, a dataset consisting of images of accessories belonging to jewelry stores in Córdoba (Spain) has been created. After testing the different image captioning architectures designed, the final model achieves a captioning accuracy of 95\%. The proposed methodology has the potential to be used in various applications such as jewelry e-commerce, inventory management or automatic jewels recognition to analyze people's tastes and social status.
Paper Structure (10 sections, 5 figures, 3 tables)

This paper contains 10 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Image captioning
  4. Computer vision for jewels
  5. Experimental design
  6. Datasets
  7. CNN and RNN architectures
  8. Experiments
  9. Results and discussion
  10. Acknowledgements

Figures (5)

  • Figure 1: Training images and captions from the final dataset.
  • Figure 2: Validating the use of Early stopping by considering only the first training epochs.
  • Figure 3: Accessories badly described by the best model. Similar shape and material.
  • Figure 4: Accessories badly described by the best model. Different materials.
  • Figure 5: Web page interface for accessory image captioning. Description for a necklace input image.