Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering

Zhixuan Shen, Haonan Luo, Sijia Li, Tianrui Li

TL;DR

An Adversarial OCR Enhancement (AOE) module is introduced, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors.

Abstract

Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems, and aggressive fine-tuning based on limited spatial location information and erroneous OCR text information often leads to inevitable overfitting. In this paper, we propose a multimodal adversarial training architecture with spatial awareness capabilities. Specifically, we introduce an Adversarial OCR Enhancement (AOE) module, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors. Simultaneously, We add a Spatial-Aware Self-Attention (SASA) mechanism to help the model better capture the spatial relationships among OCR tokens. Various experiments demonstrate that our method achieves significant performance improvements on both the ST-VQA and TextVQA datasets and provides a novel paradigm for multimodal adversarial training.

Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering

TL;DR

An Adversarial OCR Enhancement (AOE) module is introduced, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors.

Abstract

Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems, and aggressive fine-tuning based on limited spatial location information and erroneous OCR text information often leads to inevitable overfitting. In this paper, we propose a multimodal adversarial training architecture with spatial awareness capabilities. Specifically, we introduce an Adversarial OCR Enhancement (AOE) module, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors. Simultaneously, We add a Spatial-Aware Self-Attention (SASA) mechanism to help the model better capture the spatial relationships among OCR tokens. Various experiments demonstrate that our method achieves significant performance improvements on both the ST-VQA and TextVQA datasets and provides a novel paradigm for multimodal adversarial training.
Paper Structure (11 sections, 13 equations, 3 figures, 3 tables)

This paper contains 11 sections, 13 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. PROPOSED METHOD
  3. Multimodal Features Embeddings
  4. Adversarial OCR Enhancement Module
  5. Multimodal Adversarial Training
  6. Experiments
  7. Experiment Settings
  8. Experiment Results
  9. Ablation Studies
  10. Conclusion
  11. Acknowledgement

Figures (3)

  • Figure 1: An overview of our ATS. (a) We extract question tokens, visual objects, OCR tokens and OCR regions from an image and a question. (b) We perform adversarial training on the Multimodal Transformer to predict answers through iterative decoding.
  • Figure 2: The architecture of Adversarial OCR Enhancement (AOE) Module.
  • Figure 3: Qualitative results on TextVQA. For better visualization, the box in each image shows the most relevant text to the question. Best viewed in zoom.