SODBench: A Large Language Model Approach to Documenting Spreadsheet Operations

TL;DR

This work formalizes Spreadsheet Operations Documentation (SOD) as translating executed spreadsheet actions into natural language explanations to improve reproducibility and collaboration. It introduces SODBench, a benchmark of 111 code-to-NL pairs derived from xwAPI actions, and evaluates five large language models across multiple NLP metrics, revealing that GPT-4o, GPT-4o-mini, and LLaMA-3.3 perform best on code-to-NL translation while architecture and prompting strategies significantly influence results. A Retrieval-Augmented Generation (RAG) pipeline is demonstrated for generating JavaScript code for spreadsheet automation, illustrating a viable route toward automated SOD in real-world tools. The findings highlight that while LLMs can produce meaningful documentation, model size alone is not sufficient, and careful prompting, fine-tuning, and integration with API context are essential for robust, auditable spreadsheet documentation with practical deployment potential.

Abstract

Numerous knowledge workers utilize spreadsheets in business, accounting, and finance. However, a lack of systematic documentation methods for spreadsheets hinders automation, collaboration, and knowledge transfer, which risks the loss of crucial institutional knowledge. This paper introduces Spreadsheet Operations Documentation (SOD), an AI task that involves generating human-readable explanations from spreadsheet operations. Many previous studies have utilized Large Language Models (LLMs) for generating spreadsheet manipulation code; however, translating that code into natural language for SOD is a less-explored area. To address this, we present a benchmark of 111 spreadsheet manipulation code snippets, each paired with a corresponding natural language summary. We evaluate five LLMs, GPT-4o, GPT-4o-mini, LLaMA-3.3-70B, Mixtral-8x7B, and Gemma2-9B, using BLEU, GLEU, ROUGE-L, and METEOR metrics. Our findings suggest that LLMs can generate accurate spreadsheet documentation, making SOD a feasible prerequisite step toward enhancing reproducibility, maintainability, and collaborative workflows in spreadsheets, although there are challenges that need to be addressed.

Figures (5)

• Figure 1: Overview of spreadsheet workflows. The blue arrows indicate the traditional approach where spreadsheet agents break down user instructions into subtasks and create executable code. In contrast, our work emphasizes the reverse process (red arrows), translating executed actions back into natural language to document and explain user operations.
• Figure 2: Data generation workflow: SheetCopilot performs 221 benchmark tasks to create output spreadsheets. xwAPI code and natural-language description pairs are extracted from correct instances, while incorrect ones are re-executed.
• Figure 3: Example of atomic-action translation to xwAPI: user actions—writing to cell A1 and copying A1:A5 to B1:B5—are shown in VBA (top-left) and Google Apps Script/JS (bottom-left), then mapped to xwAPI equivalents (right). Each atomic operation (Write, CopyPaste) is preserved, demonstrating code conversion into xwAPI through simple syntax substitution.
• Figure 4: Variation of evaluation metrics with the number of few-shot examples
• Figure 5: Retrieval Augmented Generation (RAG) pipeline used to generate Excel manipulation JavaScript code using LangChain