Assessing Task-based Chatbots: Snapshot and Curated Datasets for Dialogflow

TL;DR

The paper tackles the lack of large, curated datasets for evaluating task-based chatbots by introducing TOFU-D, a snapshot of 1,788 Dialogflow chatbots from GitHub, and COD, a curated subset of 185 chatbots. It presents a fully automated methodology for extracting and cleaning the data and demonstrates a cross-platform validation by testing with Botium and Bandit. The results reveal gaps in test coverage and multiple security vulnerabilities, including Dialogflow-specific misconfigurations and potential code execution risks, motivating multi-platform QA research. The work provides a data-rich benchmark and a reproducible toolchain to enable rigorous quality and security assessment of real-world Dialogflow chatbots.

Abstract

In recent years, chatbots have gained widespread adoption thanks to their ability to assist users at any time and across diverse domains. However, the lack of large-scale curated datasets limits research on their quality and reliability. This paper presents TOFU-D, a snapshot of 1,788 Dialogflow chatbots from GitHub, and COD, a curated subset of TOFU-D including 185 validated chatbots. The two datasets capture a wide range of domains, languages, and implementation patterns, offering a sound basis for empirical studies on chatbot quality and security. A preliminary assessment using the Botium testing framework and the Bandit static analyzer revealed gaps in test coverage and frequent security vulnerabilities in several chatbots, highlighting the need for systematic, multi-Platform research on chatbot quality and security.

Figures (5)

• Figure 1: Methodology for the creation of the TOFU-D dataset.
• Figure 2: Methodology for the creation of the COD dataset.
• Figure 3: Complexity of TOFU-D and COD.
• Figure 4: Programming languages in TOFU-D and COD.
• Figure 5: Conversational languages in TOFU-D.