An Adaptive Benchmark for Modeling User Exploration of Large Datasets

TL;DR

SIMBA introduces an adaptive, simulation-based benchmark for evaluating DBMSs under interactive data exploration using arbitrary dashboards. It couples a goal-algebra for expressing analytic intents with an interaction-to-SQL translation and a goal-driven exploration engine, enabling realistic open-ended and targeted exploration sessions. The benchmark demonstrates that dashboard design and exploration goals materially influence DBMS performance, offering richer workloads than existing benchmarks like IDEBench. A user study and comparative experiments show SIMBA can generate plausible, human-like exploration logs and reveal performance gaps across systems, making it a valuable tool for evaluating EVA workloads and dashboard-enabled analytics platforms.

Abstract

In this paper, we present a new DBMS performance benchmark that can simulate user exploration with any specified dashboard design made of standard visualization and interaction components. The distinguishing feature of our SImulation-BAsed (or SIMBA) benchmark is its ability to model user analysis goals as a set of SQL queries to be generated through a valid sequence of user interactions, as well as measure the completion of analysis goals by testing for equivalence between the user's previous queries and their goal queries. In this way, the SIMBA benchmark can simulate how an analyst opportunistically searches for interesting insights at the beginning of an exploration session and eventually hones in on specific goals towards the end. To demonstrate the versatility of the SIMBA benchmark, we use it to test the performance of four DBMSs with six different dashboard specifications and compare our results with IDEBench. Our results show how goal-driven simulation can reveal gaps in DBMS performance missed by existing benchmarking methods and across a range of data exploration scenarios.

Figures (9)

• Figure 1: An example dashboard from Tableau Public that enables managers to monitor call centers and respond to issues in employee performance such as abandoned calls.
• Figure 2: (A) Shows how relationships between dashboard visualizations and interaction widgets are captured in the SIMBA graph layer. Here, the checkbox widget of the Customer Service Dashboard can trigger updates in the main visualizations in green and summary statistics in blue. These updates trigger new SQL queries in the data layer to be executed by the DBMS, shown in (B) and (C). Finally, (D) shows all of the relationships in the graph layer for the customer service dashboard.
• Figure 3: The Analyzing Spread template populated for the Customer Service dashboard. In this case the goal query is not syntactically achievable, but it is semantically achievable as the union of four queries generated by the Lost Calls visualization and Queue Checkbox interaction widget. Therefore, the parameterized goal can be achieved if all four queries are generated, in any order, during an exploration session.
• Figure 4: An example workflow, i.e., interaction sequence, on a sub-graph from the Customer Service dashboard and the Analyzing Spread template from \ref{['fig:analyzing_spread']}. As the user selects various queues, the properties of the connected nodes are updated and new queries are generated. In this case, the goal is achieved, i.e., the goal queries are covered, in four interactions.
• Figure 5: We use exponential decay to shift simulations from being exploration- (Markov model) to goal-focused (Oracle model). At the dotted line, both models are equally likely.