Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Data-Resilient Multi-Modal Hierarchical Multi-Label Book Genre Identification

Utsav Kumar Nareti, Soumi Chattopadhyay, Prolay Mallick, Suraj Kumar, Chandranath Adak, Ayush Vikas Daga, Adarsh Wase, Arjab Roy

TL;DR

This work tackles fine-grained book genre identification by integrating cover images, cover-text OCR, blurbs, and metadata within a hierarchical, multi-label framework. IMAGINE introduces a selective gating module to route inputs to the most informative modality and employs a two-level taxonomy (fiction vs. non-fiction, then fine-grained genres) with modality-specific and shared classifiers. The approach leverages Swin transformers for visuals, XLNet for text, and TransD-embeddings for metadata, achieving state-of-the-art performance and robustness to missing modalities on a large, expert-annotated dataset. The findings demonstrate the advantage of explicit hierarchy and adaptive fusion for structured genre prediction, with practical implications for improved recommendations and cataloging in digital libraries.

Abstract

Identifying fine-grained book genres is essential for enhancing user experience through efficient discovery, personalized recommendations, and improved reader engagement. At the same time, it provides publishers and marketers with valuable insights into consumer preferences and emerging market trends. While traditional genre classification methods predominantly rely on textual reviews or content analysis, the integration of additional modalities, such as book covers, blurbs, and metadata, offers richer contextual cues. However, the effectiveness of such multi-modal systems is often hindered by incomplete, noisy, or missing data across modalities. To address this, we propose IMAGINE (Intelligent Multi-modal Adaptive Genre Identification NEtwork), a framework designed to leverage multi-modal data while remaining robust to missing or unreliable information. IMAGINE learns modality-specific feature representations and adaptively prioritizes the most informative sources available at inference time. It further employs a hierarchical classification strategy, grounded in a curated taxonomy of book genres, to capture inter-genre relationships and support multi-label assignments reflective of real-world literary diversity. A key strength of IMAGINE is its adaptability: it maintains high predictive performance even when one modality, such as text or image, is unavailable. We also curated a large-scale hierarchical dataset that structures book genres into multiple levels of granularity, allowing for a more comprehensive evaluation. Experimental results demonstrate that IMAGINE outperformed strong baselines in various settings, with significant gains in scenarios involving incomplete modality-specific data.

Adaptive Data-Resilient Multi-Modal Hierarchical Multi-Label Book Genre Identification

TL;DR

This work tackles fine-grained book genre identification by integrating cover images, cover-text OCR, blurbs, and metadata within a hierarchical, multi-label framework. IMAGINE introduces a selective gating module to route inputs to the most informative modality and employs a two-level taxonomy (fiction vs. non-fiction, then fine-grained genres) with modality-specific and shared classifiers. The approach leverages Swin transformers for visuals, XLNet for text, and TransD-embeddings for metadata, achieving state-of-the-art performance and robustness to missing modalities on a large, expert-annotated dataset. The findings demonstrate the advantage of explicit hierarchy and adaptive fusion for structured genre prediction, with practical implications for improved recommendations and cataloging in digital libraries.

Abstract

Identifying fine-grained book genres is essential for enhancing user experience through efficient discovery, personalized recommendations, and improved reader engagement. At the same time, it provides publishers and marketers with valuable insights into consumer preferences and emerging market trends. While traditional genre classification methods predominantly rely on textual reviews or content analysis, the integration of additional modalities, such as book covers, blurbs, and metadata, offers richer contextual cues. However, the effectiveness of such multi-modal systems is often hindered by incomplete, noisy, or missing data across modalities. To address this, we propose IMAGINE (Intelligent Multi-modal Adaptive Genre Identification NEtwork), a framework designed to leverage multi-modal data while remaining robust to missing or unreliable information. IMAGINE learns modality-specific feature representations and adaptively prioritizes the most informative sources available at inference time. It further employs a hierarchical classification strategy, grounded in a curated taxonomy of book genres, to capture inter-genre relationships and support multi-label assignments reflective of real-world literary diversity. A key strength of IMAGINE is its adaptability: it maintains high predictive performance even when one modality, such as text or image, is unavailable. We also curated a large-scale hierarchical dataset that structures book genres into multiple levels of granularity, allowing for a more comprehensive evaluation. Experimental results demonstrate that IMAGINE outperformed strong baselines in various settings, with significant gains in scenarios involving incomplete modality-specific data.
Paper Structure (25 sections, 9 equations, 5 figures, 4 tables)

This paper contains 25 sections, 9 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Solution Architecture
  5. SGM: Selective Gating Module ($\Phi_S$)
  6. MIS: Multi-modal Inference Sub-Architecture ($\psi_M$)
  7. Level-1 Binary Classifier ($\Phi_B$)
  8. Level-2 Multi-label Classifier ($\Phi_M$)
  9. VIS: Visual Inference Sub-Architecture ($\psi_V$)
  10. TIS: Textual Inference Sub-Architecture ($\psi_T$)
  11. Overall Loss Function
  12. Feature Extractor
  13. Visual Feature Extractor from Cover Image
  14. Textual Feature Extractor from Blurb
  15. Textual Feature Extractor from Cover Text
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Examples of content discrepancies: (a) Uninformative blurb, (b) Irrelevant blurb, (c) Minimal visual cues on cover
  • Figure 2: Overview of our framework IMAGINE for hierarchical book genre prediction
  • Figure 4: Impact of fusions $f_1$, $f_2$, and $f_3$: (a) Level-1 ($f_1$), (b) Level-2 ($f_2$, $f_3$). $\oplus$ : addition, $\odot$ : self-attention, $\otimes$ : cross-attention, $\ominus$ : concatenation. F: Fiction, NF: Non-fiction.
  • Figure 5: Genre-wise performance analysis
  • Figure : Performance comparison of IMAGINE with baseline (a)-(e).