NeuralSI: Neural Design of Semantic Interaction for Interactive Deep Learning

TL;DR

NeuralSI proposes an end-to-end neural framework that fuses the analytic model and the visualization method for semantic interaction in interactive deep learning. By replacing traditional dimensionality reduction with a neural projection head attached to a pretrained backbone (such as BERT), the system updates via backpropagation in the same training loop, enabling out-of-sample extensions, stability, and real-time inference. Through a COVID-19 case study and simulation-based evaluations on SST, Vispubdata, and 20 Newsgroups, NeuralSI achieves comparable inference accuracy to the state-of-the-art DeepSI while delivering substantially faster time performance, especially on larger datasets. The work demonstrates that a neural integration of visualization and modeling can enhance interactive sensemaking and points to broader opportunities for end-to-end neural visual analytics frameworks.

Abstract

An increasing number of studies have utilized interactive deep learning as the analytic model of visual analytics systems for complex sensemaking tasks. In these systems, traditional interactive dimensionality reduction (DR) models are commonly utilized to build a bi-directional bridge between high-dimensional deep learning representations and low-dimensional visualizations. While these systems better capture analysts' intents in the context of human-in-the-loop interactive deep learning, traditional DR cannot support several desired properties for visual analytics, including out-of-sample extensions, stability, and real-time inference. To avoid this issue, we propose the neural design framework of semantic interaction for interactive deep learning. In our framework, we replace the traditional DR with a neural projection network and append it to the deep learning model as the task-specific output layer. Therefore, the analytic model (deep learning) and visualization method (interactive DR) form one integrated end-to-end trainable deep neural network. In order to understand the performance of the neural design in comparison to the state-of-the-art, we systematically performed two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment. The results of these studies indicate that the neural design can give semantic interaction systems substantial advantages while still keeping comparable inference ability compared to the state-of-the-art model.

Figures (9)

• Figure 1: Screenshots during the analysis of COVID-19 research articles about four risk factors (depicted in different colors) using DeepSI, an SI system with interactive DL: (1) the initial layout of all articles projected from pretrained DL representations of the raw text data; (2) the analyst performs semantic interactions to provide visual feedback regarding articles about different risk factors; these interactions are then exploited to tune the underlying DL model via interactive DR; (3) the resulting projection updated by the tuned DL. (adapted from DeepSI 10.1145/3397481.3450670)
• Figure 2: SI pipeline that follows the standard VA framework (adapted from sacha2016visualendert2012semantic). The interactive DR component serves as the visualization method responsible for capturing the intents behind analysts' interactions and updating the projection in response to the latest data relevance updated by the metric learning method. The metric learning method 10.1109/VAST.2012.6400486 is the analytic model responsible for inferring the intents from interactive DR by recalculating the data relevance and providing the updated data relevance as feedback.
• Figure 3: $\text{DeepSI}$ framework: embedding DL within the standard SI pipeline as the analytic model (adapted from Sacha_Knowledge_201410.1145/3397481.3450670). The interactive DR serves as the visualization component. The usage of traditional DR models makes the visualization component separate from the analytic model. During the co-learning process, DL and DR are in two different training loops and are required to compute separately.
• Figure 4: The NeuralSI framework uses a deep neural network as the analytic model and a relatively shallow neural network projection as the visualization method. These two components (the analytic model and visualization) form one integrated end-to-end trainable neural network responsible for concept learning and visualization generation. The two components are in the same training loop and compute together through backpropagation.
• Figure 5: Screenshots during the case study using NeuralSI: Frame 1 is the initial layout of all articles predicted by the integrated network; Frame 2 shows the similar interactions performed by the analyst in Fig. \ref{['fig:deepsi-covid']}-2. Frame 3 shows the resulting projection updated by NeuralSI.