Natural Language Interaction for Editing Visual Knowledge Graphs

TL;DR

This paper investigates natural language interaction as a method for editing visual knowledge graphs, comparing GUI editing, structured textual commands, and free-form NL inputs. Through an online user study using GQA-derived graphs across varying sizes and accuracy scenarios, NL editing demonstrated faster and more flexible edits than traditional GUI or command-based methods. The authors introduce two efficiency metrics (changes per time and changes per action) and find NL approaches to excel, particularly for larger or more complex edits, while noting design-related biases and study limitations. The work highlights the potential of NL interfaces to enhance human-in-the-loop data maintenance for knowledge graphs and related visualization tasks, suggesting directions for scaling to larger graphs and alternate visual representations.

Abstract

Knowledge graphs are often visualized using node-link diagrams that reveal relationships and structure. In many applications using graphs, it is desirable to allow users to edit graphs to ensure data accuracy or provides updates. Commonly in graph visualization, users can interact directly with the visual elements by clicking and typing updates to specific items through traditional interaction methods in the graphical user interface. However, it can become tedious to make many updates due to the need to individually select and change numerous items in a graph. Our research investigates natural language input as an alternative method for editing network graphs. We present a user study comparing GUI graph editing with two natural language alternatives to contribute novel empirical data of the trade-offs of the different interaction methods. The findings show natural language methods to be significantly more effective than traditional GUI interaction.

Figures (7)

• Figure 1: Screenshot of the interface visualizing relationships from the ConceptNet speer2017conceptnet dataset.
• Figure 2: Screenshots of the interface before and after user commands: (A) GUI Interaction via clicking and menus, (B) Textual Command using structured input, (C) Natural Language using free-form input.
• Figure 3: Screenshots of the study interface, including the text box for the Natural Language method. In each trial, participants updated the graph to match the image on the left.
• Figure 4: Average changes per time. Natural Language is significantly faster than all methods, and Textual Command outperforms GUI Interaction except in empty graph.
• Figure 5: ANOVA and Posthoc Tukey HSD test results for Interaction Method differences on changes per time (* indicates a statistically significant difference at $p<0.05$)