Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understanding

Anmol Gulati, Sahil Sen, Waqar Sarguroh, Kevin Paul

TL;DR

This work addresses the challenge of reasoning over large, multimodal enterprise spreadsheets by introducing FRTR, a retrieval-first multimodal framework that decomposes workbooks into granular row, column, and block units and uses hybrid lexical–dense retrieval with Reciprocal Rank Fusion to surface minimal, relevant evidence. FRTR integrates joint text–vision embeddings to reason across numerical data and embedded visuals, and operates through a Retrieve–Verify–Compose pipeline that outputs structured, provenance-grounded reasoning without modifying source spreadsheets. To enable reproducible evaluation, the authors release FRTR-Bench, a large-scale benchmark with 30 workbooks, nearly four million cells, and over fifty images, along with two complementary experiments against SpreadsheetLLM that demonstrate superior scalability, interpretability, and token efficiency for retrieval-based reasoning. Overall, FRTR establishes a scalable, auditable approach to multimodal spreadsheet reasoning with strong performance across diverse LLMs and practical implications for finance, auditing, and operations.

Abstract

Large Language Models (LLMs) struggle to reason over large-scale enterprise spreadsheets containing thousands of numeric rows, multiple linked sheets, and embedded visual content such as charts and receipts. Prior state-of-the-art spreadsheet reasoning approaches typically rely on single-sheet compression or full-context encoding, which limits scalability and fails to reflect how real users interact with complex, multimodal workbooks. We introduce FRTR-Bench, the first large-scale benchmark for multimodal spreadsheet reasoning, comprising 30 enterprise-grade Excel workbooks spanning nearly four million cells and more than 50 embedded images. To address these challenges, we present From Rows to Reasoning (FRTR), an advanced, multimodal retrieval-augmented generation framework that decomposes Excel workbooks into granular row, column, and block embeddings, employs hybrid lexical-dense retrieval with Reciprocal Rank Fusion (RRF), and integrates multimodal embeddings to reason over both numerical and visual information. We tested FRTR on six LLMs, achieving 74% answer accuracy on FRTR-Bench with Claude Sonnet 4.5, a substantial improvement over prior state-of-the-art approaches that reached only 24%. On the SpreadsheetLLM benchmark, FRTR achieved 87% accuracy with GPT-5 while reducing token usage by roughly 50% compared to context-compression methods.

From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understanding

TL;DR

This work addresses the challenge of reasoning over large, multimodal enterprise spreadsheets by introducing FRTR, a retrieval-first multimodal framework that decomposes workbooks into granular row, column, and block units and uses hybrid lexical–dense retrieval with Reciprocal Rank Fusion to surface minimal, relevant evidence. FRTR integrates joint text–vision embeddings to reason across numerical data and embedded visuals, and operates through a Retrieve–Verify–Compose pipeline that outputs structured, provenance-grounded reasoning without modifying source spreadsheets. To enable reproducible evaluation, the authors release FRTR-Bench, a large-scale benchmark with 30 workbooks, nearly four million cells, and over fifty images, along with two complementary experiments against SpreadsheetLLM that demonstrate superior scalability, interpretability, and token efficiency for retrieval-based reasoning. Overall, FRTR establishes a scalable, auditable approach to multimodal spreadsheet reasoning with strong performance across diverse LLMs and practical implications for finance, auditing, and operations.

Abstract

Large Language Models (LLMs) struggle to reason over large-scale enterprise spreadsheets containing thousands of numeric rows, multiple linked sheets, and embedded visual content such as charts and receipts. Prior state-of-the-art spreadsheet reasoning approaches typically rely on single-sheet compression or full-context encoding, which limits scalability and fails to reflect how real users interact with complex, multimodal workbooks. We introduce FRTR-Bench, the first large-scale benchmark for multimodal spreadsheet reasoning, comprising 30 enterprise-grade Excel workbooks spanning nearly four million cells and more than 50 embedded images. To address these challenges, we present From Rows to Reasoning (FRTR), an advanced, multimodal retrieval-augmented generation framework that decomposes Excel workbooks into granular row, column, and block embeddings, employs hybrid lexical-dense retrieval with Reciprocal Rank Fusion (RRF), and integrates multimodal embeddings to reason over both numerical and visual information. We tested FRTR on six LLMs, achieving 74% answer accuracy on FRTR-Bench with Claude Sonnet 4.5, a substantial improvement over prior state-of-the-art approaches that reached only 24%. On the SpreadsheetLLM benchmark, FRTR achieved 87% accuracy with GPT-5 while reducing token usage by roughly 50% compared to context-compression methods.
Paper Structure (36 sections, 1 equation, 3 figures, 4 tables, 1 algorithm)

This paper contains 36 sections, 1 equation, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Tables and Spreadsheets
  4. Long Context, Retrieval, and Hybrid Search
  5. Multimodal Spreadsheet Understanding and Document VQA
  6. Benchmarks for Spreadsheet Interaction
  7. Framework Overview
  8. Motivation and Design Goals
  9. Architecture Overview
  10. Retrieval and Fusion Layer
  11. Multimodal Embedding Layer
  12. Reasoning and Output Generation
  13. FRTR-Bench: Benchmark for Spreadsheet Reasoning
  14. Dataset Construction
  15. Evaluation Protocol
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Excel workbooks are decomposed into rows, columns, sliding context windows, and embedded images during indexing. These components are stored in a multimodal hybrid vector database supporting lexical and dense fields. Hybrid lexical–dense retrieval (RRF) fetches relevant evidence for LLM reasoning.
  • Figure 2: Performance comparison by difficulty level across methods on FRTR-Bench.
  • Figure 3: Visualization of prompt token usage for FRTR and SpreadsheetLLM.