RV4Chatbot: Are Chatbots Allowed to Dream of Electric Sheep?

TL;DR

RV4Chatbot, a Runtime Verification framework designed to monitor deviations in chatbot behaviour, is introduced and expected behaviours as interaction protocols between the user and the chatbot are formalised.

Abstract

Chatbots have become integral to various application domains, including those with safety-critical considerations. As a result, there is a pressing need for methods that ensure chatbots consistently adhere to expected, safe behaviours. In this paper, we introduce RV4Chatbot, a Runtime Verification framework designed to monitor deviations in chatbot behaviour. We formalise expected behaviours as interaction protocols between the user and the chatbot. We present the RV4Chatbot design and describe two implementations that instantiate it: RV4Rasa, for monitoring chatbots created with the Rasa framework, and RV4Dialogflow, for monitoring Dialogflow chatbots. Additionally, we detail experiments conducted in a factory automation scenario using both RV4Rasa and RV4Dialogflow.

Figures (6)

• Figure 1: RV4Chatbot architecture.
• Figure 2: The RV4Rasa instantiation of RV4Chatbot.
• Figure 3: The RV4Dialogflow instantiation of RV4Chatbot.
• Figure 4: RV4Dialogflow policy flow.
• Figure 5: Initial scenario of the simulated factory floor (above) and the result after further iterations of adding new objects in the scene (below) taken from DBLP:journals/robotics/GattiM23.