Breathing New Life into Existing Visualizations: A Natural Language-Driven Manipulation Framework

TL;DR

The paper introduces a natural language–driven framework to manipulate existing visualizations by translating user queries into a hierarchical set of visualization tasks and then into a sequence of atomic manipulations performed in situ. It defines a four-level, seven-type manipulation space and a hierarchical task design, powered by a two-stage NL–task translator and manipulation parser, with a knowledge-distillation training approach that uses a large LLM to curate data for a smaller T5 model. A cross-domain NL–task dataset (486 attribute combinations, 65 topics, 5,867 pairs) and extensive evaluation demonstrate high parsing accuracy and effective, smooth visualization transformations validated through a user study. The framework enables flexible, intuitive, in-situ interaction with diverse visualizations, offering practical benefits for data exploration and interpretation.

Abstract

We propose an approach to manipulate existing interactive visualizations to answer users' natural language queries. We analyze the natural language tasks and propose a design space of a hierarchical task structure, which allows for a systematic decomposition of complex queries. We introduce a four-level visualization manipulation space to facilitate in-situ manipulations for visualizations, enabling a fine-grained control over the visualization elements. Our methods comprise two essential components: the natural language-to-task translator and the visualization manipulation parser. The natural language-to-task translator employs advanced NLP techniques to extract structured, hierarchical tasks from natural language queries, even those with varying degrees of ambiguity. The visualization manipulation parser leverages the hierarchical task structure to streamline these tasks into a sequence of atomic visualization manipulations. To illustrate the effectiveness of our approach, we provide real-world examples and experimental results. The evaluation highlights the precision of our natural language parsing capabilities and underscores the smooth transformation of visualization manipulations.

Figures (9)

• Figure 1: NL-Task translator, fine-tuned on a large language model and a constructed multi-domain and diverse NL-Task dataset, can transform a natural language query input into a hierarchical structure of tasks. The tasks are then transformed into a series of visualization manipulations by the visualization manipulation parser. Finally, the visualization manipulation parser changes the visualization in situ to respond to the natural language query.
• Figure 2: Natural language queries in the context of visualization are transformed into nested high-level tasks, which are represented through a series of visualization manipulations. Visualization manipulations parse tasks from bottom to top, beginning with the resolution of filtering conditions and followed by comparative tasks. Different tasks are represented through the combination of visualization manipulations such as highlighting, annotation, reordering, or remapping.
• Figure 3: Design space of the natural language task.
• Figure 4: The data creation process ensures diversity in three dimensions, namely data attributes, tasks, and natural language expressions.
• Figure 5: (a) The encoder-decoder model of the large language model, text-to-text translate transformer (T5 raffel2020exploring). (b) The loss variation during the training process can be observed, showing that the model gradually converged over 30 epochs.