KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts

TL;DR

KnowledgeVIS, a human-in-the-loop visual analytics system for interpreting language models using fill-in-the-blank sentences as prompts, reveals learned associations that intuitively connect what language models learn during training to natural language tasks downstream.

Abstract

Recent growth in the popularity of large language models has led to their increased usage for summarizing, predicting, and generating text, making it vital to help researchers and engineers understand how and why they work. We present KnowledgeVis, a human-in-the-loop visual analytics system for interpreting language models using fill-in-the-blank sentences as prompts. By comparing predictions between sentences, KnowledgeVis reveals learned associations that intuitively connect what language models learn during training to natural language tasks downstream, helping users create and test multiple prompt variations, analyze predicted words using a novel semantic clustering technique, and discover insights using interactive visualizations. Collectively, these visualizations help users identify the likelihood and uniqueness of individual predictions, compare sets of predictions between prompts, and summarize patterns and relationships between predictions across all prompts. We demonstrate the capabilities of KnowledgeVis with feedback from six NLP experts as well as three different use cases: (1) probing biomedical knowledge in two domain-adapted models; and (2) evaluating harmful identity stereotypes and (3) discovering facts and relationships between three general-purpose models.

Figures (10)

• Figure 1: KnowledgeVIS integrates multiple views to reveal associations that LLMs learn during training. Above, a user investigates whether BERT exhibits associations that reveal learned conceptual relationships (Sect. \ref{['sec:case_knowledge']}), helping them interpret how BERT works.
• Figure 2: The Set View showing our variation on the parallel tag cloud layout, a stepwise degree of interest list based on fisheye menus Bederson:2000:FisheyeMenus for a selected word when sorting by rank, described in Sect. \ref{['sec:set_view']}.
• Figure 3: Data sets and prompts used in our use cases (Sect. \ref{['sec:cases_evaluation']}) to show how KnowledgeVIS can help NLP researchers and engineers interpret LLMs.
• Figure 4: The Set View showing our variation on the parallel tag cloud layout, a step-wise degree of interest list based on fisheye menus Bederson:2000:FisheyeMenus for a selected word when sorting by rank, described in Fig. \ref{['sec:set_view']}. We query the top $k=16$ predictions and select "cook" from the resulting view. Our user can quickly see "cook" is ranked lower for the "man"/"boy" subjects than for "woman"/"girl", showing a slight gender bias in the probability of the occupation occurring, even though it appears for all subjects.
• Figure 5: Two Heat Maps showing how grammar and phrasing affect PubMedBERT. The glyphs highlight predictions mentioned in the body text.