Vision Language Models for Spreadsheet Understanding: Challenges and Opportunities

TL;DR

This work systematically probes Vision Language Models for spreadsheet understanding by isolating OCR, spatial positioning, and visual format recognition through self-supervised spreadsheet-to-image tasks. It introduces a boundary-based approach for spreadsheet table detection and explores three image settings (column width, style, address augmentation) to reveal how visual signals affect model performance. Empirical results show strong OCR capabilities but limited spatial-perception and format-recognition skills, with table-detection still lagging behind CNN-based methods like TableSense. The paper highlights data-generation strategies and prompts as practical routes to scale VLM understanding of structured, grid-like data in spreadsheets, suggesting directions for future improvements.

Abstract

This paper explores capabilities of Vision Language Models on spreadsheet comprehension. We propose three self-supervised challenges with corresponding evaluation metrics to comprehensively evaluate VLMs on Optical Character Recognition (OCR), spatial perception, and visual format recognition. Additionally, we utilize the spreadsheet table detection task to assess the overall performance of VLMs by integrating these challenges. To probe VLMs more finely, we propose three spreadsheet-to-image settings: column width adjustment, style change, and address augmentation. We propose variants of prompts to address the above tasks in different settings. Notably, to leverage the strengths of VLMs in understanding text rather than two-dimensional positioning, we propose to decode cell values on the four boundaries of the table in spreadsheet boundary detection. Our findings reveal that VLMs demonstrate promising OCR capabilities but produce unsatisfactory results due to cell omission and misalignment, and they notably exhibit insufficient spatial and format recognition skills, motivating future work to enhance VLMs' spreadsheet data comprehension capabilities using our methods to generate extensive spreadsheet-image pairs in various settings.

Figures (14)

• Figure 1: A sample spreadsheet showing various challenging points in spreadsheet understanding task.
• Figure 2: Illustration of spreadsheet-to-image settings.
• Figure 3: The prompt of OCR task.
• Figure 4: The difference between LCS matching and Strict matching.
• Figure 5: The prompt for vanilla experiment of spatial position perception task.