A Preliminary Roadmap for LLMs as Assistants in Exploring, Analyzing, and Visualizing Knowledge Graphs

TL;DR

This paper addresses how large language models can act as assistants in exploring, analyzing, and visualizing knowledge graphs. It employs a mixed-methods study with 20 KG and LLM professionals to elicit tasks, interaction modalities, and potential pitfalls, and uses thematic analysis to derive a preliminary design roadmap. The findings show strong demand for LLM-assisted data retrieval, multi-turn question refinement, and on-demand visualizations, alongside cautions about semantic intent, hallucinations, and prompt engineering. The work contributes a practical roadmap and design considerations for LLM-driven KG exploration systems, highlighting opportunities for collaborative workflows and integrated visual analytics with real-world impact for researchers and practitioners.

Abstract

We present a mixed-methods study to explore how large language models (LLMs) can assist users in the visual exploration and analysis of knowledge graphs (KGs). We surveyed and interviewed 20 professionals from industry, government laboratories, and academia who regularly work with KGs and LLMs, either collaboratively or concurrently. Our findings show that participants overwhelmingly want an LLM to facilitate data retrieval from KGs through joint query construction, to identify interesting relationships in the KG through multi-turn conversation, and to create on-demand visualizations from the KG that enhance their trust in the LLM's outputs. To interact with an LLM, participants strongly prefer a chat-based 'widget,' built on top of their regular analysis workflows, with the ability to guide the LLM using their interactions with a visualization. When viewing an LLM's outputs, participants similarly prefer a combination of annotated visuals (e.g., subgraphs or tables extracted from the KG) alongside summarizing text. However, participants also expressed concerns about an LLM's ability to maintain semantic intent when translating natural language questions into KG queries, the risk of an LLM 'hallucinating' false data from the KG, and the difficulties of engineering a 'perfect prompt.' From the analysis of our interviews, we contribute a preliminary roadmap for the design of LLM-driven knowledge graph exploration systems and outline future opportunities in this emergent design space.

Figures (4)

• Figure 1: Distribution of responses from participants on which tasks or analytic goals they want an LLM to help facilitate when using a KG. We discuss each of these tasks in Section \ref{['sec:tasks']}.
• Figure 2: A wireframe design, informed by participants' responses (Section \ref{['sec:justify']}), showing an LLM's outputs with built-in justification. On the left side, the user asks about connections between two nodes in the KG. The LLM responds with a list of target entities that connect the two nodes. On the right, the subgraph visualization from the KG shows the simplified paths between the nodes. When the user clicks on Node X, the panel on the bottom shows the original data source. Participants remarked that this type of design could help them examine true connections in the source data, thus making them more aware of the validity of an LLM's responses. We discuss the workflow for this system in Section \ref{['sec:workflow']}.
• Figure 3: Responses from participants on how they would prefer to interface with the LLM (top) and how they would prefer the LLM to return its results when interfacing with the KG (bottom). We discuss each of these modalities in Section \ref{['sec:design-considerations']}.
• Figure 4: Results of our participants' responses on what weaknesses an LLM might hold when helping users interface with a knowledge graph. We discuss further in Section \ref{['sec:weaknesses']}.