Into the Void: Mapping the Unseen Gaps in High Dimensional Data

TL;DR

The paper tackles the challenge of discovering valuable, unseen configurations in high-dimensional spaces by introducing ESA, a scalable, physics-inspired search that identifies empty-space configurations. Integrated with GapMiner, a visual analytics platform enabling human-in-the-loop refinement and a DNN-assisted autonomous search, the approach progressively shifts responsibility from humans to AI as the model trains. The authors demonstrate greater discovery of high-quality configurations across multiple domains—system optimization, wine quality, adversarial learning, and reinforcement learning—compared with random or baseline methods, supported by user studies and case studies. This framework offers a practical pipeline for efficient exploration of sparse parameter spaces, with potential impact on design optimization, security, and AI-driven experimentation.

Abstract

We present a comprehensive pipeline, augmented by a visual analytics system named ``GapMiner'', that is aimed at exploring and exploiting untapped opportunities within the empty areas of high-dimensional datasets. Our approach begins with an initial dataset and then uses a novel Empty Space Search Algorithm (ESA) to identify the center points of these uncharted voids, which are regarded as reservoirs containing potentially valuable novel configurations. Initially, this process is guided by user interactions facilitated by GapMiner. GapMiner visualizes the Empty Space Configurations (ESC) identified by the search within the context of the data, enabling domain experts to explore and adjust ESCs using a linked parallel-coordinate display. These interactions enhance the dataset and contribute to the iterative training of a connected deep neural network (DNN). As the DNN trains, it gradually assumes the task of identifying high-potential ESCs, diminishing the need for direct user involvement. Ultimately, once the DNN achieves adequate accuracy, it autonomously guides the exploration of optimal configurations by predicting performance and refining configurations, using a combination of gradient ascent and improved empty-space searches. Domain users were actively engaged throughout the development of our system. Our findings demonstrate that our methodology consistently produces substantially superior novel configurations compared to conventional randomization-based methods. We illustrate the effectiveness of our method through several case studies addressing various objectives, including parameter optimization, adversarial learning, and reinforcement learning.

Figures (18)

• Figure 1: Our three-phase workflow.
• Figure 2: An example of Lennard-Jones Potential. The $x$ axis is the distance between particles ($r$) and the y axis is the outcome $V(r)$. The potential is positive when the particle distance is less than $\sigma$, indicating a repulsive force, and negative for larger distances, where there is an attractive force. The minimum potential (strongest attraction) is $-\epsilon$.
• Figure 3: Given 300 random samples (blue) and 600 random agents (red) in 2D space, we show the results of ESA (a) without and (b) with momentum.
• Figure 4: GapMiner visual interface where a selected ESC is reflected in all displays. (A) Control Panel. From top to bottom: (a) File Selector to load a dataset of initial verified configurations with values for all parameter variables. (b) Target Variable Configurator with an interface for breaking its value range into discrete intervals. (c) Empty Space Algorithm (ESA) Configurator to select the ESA and a slider to set the ESC batch size. (d) Empty Space Configuration (ESC) Range Selector to control which target variable intervals are used for display and ESC proposals. (e) Overview Quality Monitor screeplot that shows the amount of data variance captured by the Overview (PCA) Display. (B) Overview (PCA) Display with data distribution contours, raw or modified ESCs rendered as points, and color legends. (C) Empty Space Configuration (ESC) Editor. From left to right: (a) Parallel Coordinate Plot where users can configure ESCs starting from a raw ESC or an existing configuration. (b) Neighbor plot of the selected ESC providing a local view of the distribution of its nearest existing configurations. (D) Progress Tracker. From top to bottom: (a) Budget/Reward Display that captures the aggregated evaluation cost and merit of the ESC exploration so far. (b) Training Status Display of the assistive DNN. (c) Pareto Frontier Plot that shows the Pareto frontiers of existing configurations (red) and ESCs (gray) with respect to two user-chosen merit (target) variables.
• Figure 5: An example Pareto front at the three key stages. (a) The initial stage when the DNN has just started training. There are only few configurations in the "existing" set, colored blue. (b) The front when the DNN has achieved $t_1$. The "existing" set has grown. (c) The front when the DNN has achieved $t_2$. The "existing" set now covers much of the front's interior.