From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysis

TL;DR

This study introduces StatZ, a conversational agent for statistical analysis, and benchmarks it against GUI-based tools (SPSS, SAS, Stata, JMP). Using a mixed-methods evaluation with 51 participants across Iris and NYC Taxi datasets, StatZ outperforms GUI tools in accuracy, task time, and user satisfaction, while reducing cognitive and motor load. The results suggest that context-aware, on-demand explanations embedded in a conversational interface can streamline statistical workflows and democratize data analysis for users with varying levels of statistical expertise. The work provides design guidelines for future statistical software and highlights the potential of conversational agents to transform data analysis practice, though it notes limitations related to sample size, task scope, and long-term use.

Abstract

The rapid proliferation of data science forced different groups of individuals with different backgrounds to adapt to statistical analysis. We hypothesize that conversational agents are better suited for statistical analysis than traditional graphical user interfaces (GUI). In this work, we propose a novel conversational agent, StatZ, for statistical analysis. We evaluate the efficacy of StatZ relative to established statistical software:SPSS, SAS, Stata, and JMP in terms of accuracy, task completion time, user experience, and user satisfaction. We combined the proposed analysis question from state-of-the-art language models with suggestions from statistical analysis experts and tested with 51 participants from diverse backgrounds. Our experimental design assessed each participant's ability to perform statistical analysis tasks using traditional statistical analysis tools with GUI and our conversational agent. Results indicate that the proposed conversational agents significantly outperform GUI statistical software in all assessed metrics, including quantitative (task completion time, accuracy, and user experience), and qualitative (user satisfaction) metrics. Our findings underscore the potential of using conversational agents to enhance statistical analysis processes, reducing cognitive load and learning curves and thereby proliferating data analysis capabilities, to individuals with limited knowledge of statistics.

Figures (6)

• Figure 1: Front-end sample for transform and scaling data. User can select the method of their choice from Min-max scaling, z-score scaling, L1 norm scaling and L2 norm scaling.
• Figure 2: Box Plot of Accuracy Across Different Software
• Figure 3: Box Plot of Time Efficiency Using Various Statistical Software
• Figure 4: Average Mouse Distance Traveled Among Statistical Tools
• Figure 5: Comparison of Keyboard and Mouse Usage Across Tools