A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models

TL;DR

This work introduces a human-centered, multi-dimensional risk metric for evaluating LLM-based chatbots that accounts for damages to the service provider, users, and third parties. The approach integrates attack complexity via $\delta_t$, industry context via $I$, and user demographics via $P$, formalized in the risk vector $R_d = (R_{hs},R_{hu},R_{ho})$ and its medians-driven variant $R^*_d$, and is implemented as a Garak wrapper with enhanced probes and a Flask API. The methodology is demonstrated on three RAG-based chatbots (Llama2-7B, Vicuna-7B, Neural Chat-7B), showing that prompt protection mitigates some, but not all, risks—especially those affecting third parties such as malware generation and scams—and that risk varies significantly with industry and age. The findings underscore the need for context-aware security evaluation tools for conversational AI and point to practical paths for short-term mitigations and long-term model-design improvements. Overall, the paper offers a repeatable, instrumented framework for operational risk assessment of AI-driven chat systems that can guide secure deployment and regulatory compliance in real-world settings.

Abstract

The emergence of Generative AI (Gen AI) and Large Language Models (LLMs) has enabled more advanced chatbots capable of human-like interactions. However, these conversational agents introduce a broader set of operational risks that extend beyond traditional cybersecurity considerations. In this work, we propose a novel, instrumented risk-assessment metric that simultaneously evaluates potential threats to three key stakeholders: the service-providing organization, end users, and third parties. Our approach incorporates the technical complexity required to induce erroneous behaviors in the chatbot--ranging from non-induced failures to advanced prompt-injection attacks--as well as contextual factors such as the target industry, user age range, and vulnerability severity. To validate our metric, we leverage Garak, an open-source framework for LLM vulnerability testing. We further enhance Garak to capture a variety of threat vectors (e.g., misinformation, code hallucinations, social engineering, and malicious code generation). Our methodology is demonstrated in a scenario involving chatbots that employ retrieval-augmented generation (RAG), showing how the aggregated risk scores guide both short-term mitigation and longer-term improvements in model design and deployment. The results underscore the importance of multi-dimensional risk assessments in operationalizing secure, reliable AI-driven conversational systems.

Figures (4)

• Figure 1: Schematic representation of various perspectives on LLM security issues, highlighting different risks, vulnerabilities, and currently available metrics.
• Figure 2: Schematic representation of the evaluation framework with the integration of the risk metric calculator over the GARAK components.
• Figure 3: Prompts Structures for the different Tested RAG systems
• Figure 4: Figure illustrating the effect of the target age group and the industry in which the chatbot operates on the values of the evaluated metric.