Talk Me Through It: Developing Effective Systems for Chart Authoring

TL;DR

This paper investigates how spoken imagined-chart instructions differ from typed imagined-chart and typed existing-chart prompts, revealing that imagined-chart data—especially in voice form—exhibits richer command formats, element specifications, and linguistic features. Through Phase I, it establishes structural distinctions across modalities and prompt types; Phase II demonstrates that an LLM fine-tuned on spoken imagined-chart data outperforms one trained on typed existing-chart data in both voice and text inputs. The findings underscore the necessity of modality-aware training and more naturalistic data collection for chart-authoring systems, and they offer design guidelines and a public imagined-chart dataset to accelerate development. Collectively, the work advances multimodal chart authoring by aligning training data with real-world prompting behavior and by evaluating cross-modal performance impacts.

Abstract

Recent chart-authoring systems increasingly focus on natural-language input, enabling users to form a mental image of the chart they wish to create and express this intent using spoken instructions (spoken imagined-chart data). Yet these systems are predominantly trained on typed instructions written while viewing the target chart (typed existing-chart data). While the cognitive processes for describing an existing chart arguably differ from those for creating a new chart, the structural differences in the corresponding prompts remain underexplored. We present empirical findings on the structural differences among spoken imagined-chart instructions, typed imagined-chart instructions, and typed existing-chart instructions for chart creation, showing that imagined-chart prompts contain richer command formats, element specifications, and complex linguistic features, especially in spoken instructions. We then compare the performance of systems trained on spoken imagined-chart data versus typed existing-chart data, finding that the first system outperforms the second one on both voice and text input, highlighting the necessity of targeted training on spoken imagined-chart data. We conclude with design guidelines for chart-authoring systems to improve performance in real-world scenarios.

Figures (6)

• Figure 1: The number of times each element of 5 major types was applied to each input strategy in spoken imagined-chart (A) and typed imagined-chart (B) chart-authoring instructions collected in Phase I, typed existing-chart instructions from the NLV Corpus Srinivasan01 (C), and synthetic typed existing-chart instructions from the nvBench Luo01 (D). Color intensity corresponds to data magnitude.
• Figure 2: Summary of system performance comparison in Phase II across four configurations: System 2 on text input (A), System 2 on voice input (B), System 1 on voice input (C), and System 1 on text input (D). Each bar represents the frequency of 5 common outcomes: chart type errors, X and Y value errors, design errors, annotation errors, or correctly generated charts (OK).
• Figure 3: Examples of the text stimuli provided to participants of user studies in Phase I. From statista.
• Figure 4: Summary of the word count per instruction of the 76 spoken imagined-chart and 65 typed imagined-chart instructions collected in Phase I, 200 typed existing-chart instructions from the NLV Corpus Srinivasan01, and 200 synthetic typed existing-chart instructions from the nvBench Luo01. The (count) next to each chart type indicates the total number of instructions associated with that type.
• Figure 5: User's view of the web-based study environment in Phase II. The interface layout has five sections: the information panel with the current system name and text stimulus (1), the chart display area with the history of provided instructions and generated charts (2), the input field for typing (3), the recording start and stop button (4), and the navigation button to clear the history and proceed to the next text stimulus (5).