Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach

Siyu Duan, Jun Wang, Qi Su

TL;DR

A novel Multimodal Multitask Restoring Model (MMRM) is proposed to restore ancient texts, particularly emphasising the ideograph, that combines context understanding with residual visual information from damaged ancient artefacts, enabling it to predict damaged characters and generate restored images simultaneously.

Abstract

Cultural heritage serves as the enduring record of human thought and history. Despite significant efforts dedicated to the preservation of cultural relics, many ancient artefacts have been ravaged irreversibly by natural deterioration and human actions. Deep learning technology has emerged as a valuable tool for restoring various kinds of cultural heritages, including ancient text restoration. Previous research has approached ancient text restoration from either visual or textual perspectives, often overlooking the potential of synergizing multimodal information. This paper proposes a novel Multimodal Multitask Restoring Model (MMRM) to restore ancient texts, particularly emphasising the ideograph. This model combines context understanding with residual visual information from damaged ancient artefacts, enabling it to predict damaged characters and generate restored images simultaneously. We tested the MMRM model through experiments conducted on both simulated datasets and authentic ancient inscriptions. The results show that the proposed method gives insightful restoration suggestions in both simulation experiments and real-world scenarios. To the best of our knowledge, this work represents the pioneering application of multimodal deep learning in ancient text restoration, which will contribute to the understanding of ancient society and culture in digital humanities fields.

Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach

TL;DR

A novel Multimodal Multitask Restoring Model (MMRM) is proposed to restore ancient texts, particularly emphasising the ideograph, that combines context understanding with residual visual information from damaged ancient artefacts, enabling it to predict damaged characters and generate restored images simultaneously.

Abstract

Cultural heritage serves as the enduring record of human thought and history. Despite significant efforts dedicated to the preservation of cultural relics, many ancient artefacts have been ravaged irreversibly by natural deterioration and human actions. Deep learning technology has emerged as a valuable tool for restoring various kinds of cultural heritages, including ancient text restoration. Previous research has approached ancient text restoration from either visual or textual perspectives, often overlooking the potential of synergizing multimodal information. This paper proposes a novel Multimodal Multitask Restoring Model (MMRM) to restore ancient texts, particularly emphasising the ideograph. This model combines context understanding with residual visual information from damaged ancient artefacts, enabling it to predict damaged characters and generate restored images simultaneously. We tested the MMRM model through experiments conducted on both simulated datasets and authentic ancient inscriptions. The results show that the proposed method gives insightful restoration suggestions in both simulation experiments and real-world scenarios. To the best of our knowledge, this work represents the pioneering application of multimodal deep learning in ancient text restoration, which will contribute to the understanding of ancient society and culture in digital humanities fields.
Paper Structure (21 sections, 9 equations, 8 figures, 4 tables)

This paper contains 21 sections, 9 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Task Define
  5. Data
  6. Damaged Text
  7. Damaged Image
  8. Model
  9. Modules
  10. Multitask Learning
  11. Curriculum Learning
  12. Experiment
  13. Baselines
  14. Proposed Approach
  15. Metrics
  16. ...and 6 more sections

Figures (8)

  • Figure 1: The process of simulating images of damaged characters. The first picture shows images of ideograms generated by 108 kinds of fonts. The second picture shows the simulated images of undamaged characters. The third picture shows the simulated images of damaged characters.
  • Figure 2: Two concrete cases of Additive Damage and Fading Damage. The left image is from a rubbing of the 'Cao Quan Stele'; the right image is from the paper book 'Sima Fa Ji Jie'.
  • Figure 3: The formulation of Multimodal Multitask Restoration Model (MMRM).
  • Figure 4: Cases from different models in simulation experiments
  • Figure 5: The full rubbing and photo of the 'Inscription of Sweet Spring in Jiucheng Palace'
  • ...and 3 more figures