Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics

TL;DR

This work tackles understanding infographics by bridging the gap between natural-image training and the unique icon-rich, text-laden content of infographics. It introduces Visually29K, a large infographic dataset with rich icon and tag annotations, and a synthetic-data pipeline that pastes transparent-background icons onto infographic patches to train a class-agnostic icon-proposal model. Using Faster R-CNN with tailored adaptations, the authors show that models trained with synthetic icon data outperform those trained on natural images in locating icons. They further demonstrate a multi-modal summarization system that combines OCR-extracted text, icon proposals, and icon classification to produce representative text tags and visual hashtags, enabling applications in knowledge retrieval, visual search, and VQA for infographics. Overall, the approach provides a scalable pathway to parse and summarize complex infographic content by jointly leveraging textual and visual cues.

Abstract

Widely used in news, business, and educational media, infographics are handcrafted to effectively communicate messages about complex and often abstract topics including `ways to conserve the environment' and `understanding the financial crisis'. Composed of stylistically and semantically diverse visual and textual elements, infographics pose new challenges for computer vision. While automatic text extraction works well on infographics, computer vision approaches trained on natural images fail to identify the stand-alone visual elements in infographics, or `icons'. To bridge this representation gap, we propose a synthetic data generation strategy: we augment background patches in infographics from our Visually29K dataset with Internet-scraped icons which we use as training data for an icon proposal mechanism. On a test set of 1K annotated infographics, icons are located with 38% precision and 34% recall (the best model trained with natural images achieves 14% precision and 7% recall). Combining our icon proposals with icon classification and text extraction, we present a multi-modal summarization application. Our application takes an infographic as input and automatically produces text tags and visual hashtags that are textually and visually representative of the infographic's topics respectively.

Figures (6)

• Figure 1: We make 3 contributions: a) We present Visually29K, a curated dataset of infographics; b) We generate synthetic data by augmenting Internet-scraped icons onto patches of infographics to train an icon proposal mechanism; c) We evaluate our automatic icon proposals and present a multi-modal summarization application that takes an infographic and outputs the text tags and visual hashtags that are most representative of the infographic's topics.
• Figure 2: Synthetic data generation pipeline. a) Icons with transparent backgrounds scraped from Google. b) Patches selected for augmenting icons, using different approaches. The approach on the left allows more overlap of icons with background elements. The approach on the right is more conservative, selecting appropriate patches to add icons to. c) Infographic windows augmented with the scraped icons.
• Figure 3: Our computational pipeline for parsing an infographic and computing a multi-modal summary. a) The output of our fully-automatic annotation system, running text detection and OCR using Google's Cloud Vision API googleText (semi-transparent green boxes), and our icon detection and classification (red outlines). We trained an icon proposal mechanism with synthetic data to make this system possible. The underlying infographic has been faded to facilitate visualization. b) Our multi-modal summarization application uses the detected text and icons on an infographic to produce the text tags and visual hashtags most representative of the infographic's topics.
• Figure 4: Visual hashtags for different concepts. We include 6 different tag classes, sorted by mAP. For each tag class, depicted are the top 4 instances with highest classifier confidence for each tag, constrained to come from different images. Also indicated is the total number (N) of icon proposals per tag class.
• Figure 5: Examples of our automated multi-modal summarization pipeline, which given an infographic as input, predicts text tags and corresponding visual hashtags. In both (a) and (b), the predicted text tags for the infographics are correct, and the predicted visual hashtags (solid blue boxes) overlap with human annotations (red boxes). Because a single tag might not be sufficient to summarize an infographic, we also provide an additional predicted text tag (second most likely) and corresponding visual hashtag for (a) and (b). In (c)-(e) the text model predicts the wrong tag. In (c), the semantic meaning of the predicted tag is preserved, so the visual hashtag is still correct. In (d) and (e), the wrong visual hashtags are returned as a result of the text predictions. However, we show that if the correct text tag would have been used (bottom, red), correct visual hashtags would have been returned. In dashed blue are all our icon proposals for each infographic. The underlying infographics have been faded to facilitate visualization.