Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChartFormer: A Large Vision Language Model for Converting Chart Images into Tactile Accessible SVGs

Omar Moured, Sara Alzalabny, Anas Osman, Thorsten Schwarz, Karin Muller, Rainer Stiefelhagen

TL;DR

This work tackles inaccessible chart visualizations by converting raster charts into tactile-accessible SVGs using ChartFormer, a transformer-based approach built on Vision-Language Model foundations. It introduces the Chart2Tactile dataset, a 10,000-sample synthetic collection of charts across four types, crafted under accessibility guidelines to enable tactile rendering. A pilot study with four blind/VI participants evaluates the generated SVGs on a HyperBraille 2D display, demonstrating feasibility while highlighting rendering artifacts (e.g., stair-stepping) and the need for broader chart-type support. The contributions—the end-to-end ChartFormer system, the tactile-focused Chart2Tactile dataset, and empirical user insights—advance accessible graphics production for education and research, with public resources to foster further development.

Abstract

Visualizations, such as charts, are crucial for interpreting complex data. However, they are often provided as raster images, which are not compatible with assistive technologies for people with blindness and visual impairments, such as embossed papers or tactile displays. At the same time, creating accessible vector graphics requires a skilled sighted person and is time-intensive. In this work, we leverage advancements in the field of chart analysis to generate tactile charts in an end-to-end manner. Our three key contributions are as follows: (1) introducing the ChartFormer model trained to convert raster chart images into tactile-accessible SVGs, (2) training this model on the Chart2Tactile dataset, a synthetic chart dataset we created following accessibility standards, and (3) evaluating the effectiveness of our SVGs through a pilot user study with an refreshable two-dimensional tactile display. Our work is publicly available at https://github.com/nsothman/ChartFormer .

ChartFormer: A Large Vision Language Model for Converting Chart Images into Tactile Accessible SVGs

TL;DR

This work tackles inaccessible chart visualizations by converting raster charts into tactile-accessible SVGs using ChartFormer, a transformer-based approach built on Vision-Language Model foundations. It introduces the Chart2Tactile dataset, a 10,000-sample synthetic collection of charts across four types, crafted under accessibility guidelines to enable tactile rendering. A pilot study with four blind/VI participants evaluates the generated SVGs on a HyperBraille 2D display, demonstrating feasibility while highlighting rendering artifacts (e.g., stair-stepping) and the need for broader chart-type support. The contributions—the end-to-end ChartFormer system, the tactile-focused Chart2Tactile dataset, and empirical user insights—advance accessible graphics production for education and research, with public resources to foster further development.

Abstract

Visualizations, such as charts, are crucial for interpreting complex data. However, they are often provided as raster images, which are not compatible with assistive technologies for people with blindness and visual impairments, such as embossed papers or tactile displays. At the same time, creating accessible vector graphics requires a skilled sighted person and is time-intensive. In this work, we leverage advancements in the field of chart analysis to generate tactile charts in an end-to-end manner. Our three key contributions are as follows: (1) introducing the ChartFormer model trained to convert raster chart images into tactile-accessible SVGs, (2) training this model on the Chart2Tactile dataset, a synthetic chart dataset we created following accessibility standards, and (3) evaluating the effectiveness of our SVGs through a pilot user study with an refreshable two-dimensional tactile display. Our work is publicly available at https://github.com/nsothman/ChartFormer .
Paper Structure (17 sections, 3 figures)

This paper contains 17 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Metadata Extraction
  4. Tactile Charts
  5. Available Datasets
  6. Chart2Tactile Dataset
  7. Metadata Collection
  8. SVG Guidelines
  9. SVG Creation
  10. ChartFormer Model
  11. Pilot User Study
  12. Procedure
  13. Results & Discussion
  14. Limitations
  15. Conclusion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: A scatter plot sample: (a) the original synthesized raster image; (b) the tactile version following accessibility guidelines.
  • Figure 2: The ChartFormer takes a raster x-y plot as an input. The essential metadata and styles are extracted, which are then used to populate the svgwrite templates. For better viewing resolution, please visit our project page.
  • Figure 3: SVG-formatted line charts used in the user study, showcasing varying complexities: (A) a single line; (B) two lines; (C) six lines. For better viewing resolution, please visit our project page.