Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Securing LLM-as-a-Service for Small Businesses: An Industry Case Study of a Distributed Chatbot Deployment Platform

Jiazhu Xie, Bowen Li, Heyu Fu, Chong Gao, Ziqi Xu, Fengling Han

TL;DR

The paper tackles the challenge of deploying LLM-based support chatbots for small businesses by presenting an open-source, multi-tenant platform built on distributed, low-cost k3s clusters with an encrypted overlay. It introduces a layered security approach against prompt injection in RAG-based chatbots, combining prompt-level guard prompts with a model-agnostic GenTel-Shield detector, deployed without model retraining. Through a real-world e-commerce case study, the work evaluates both security effectiveness and deployment efficiency, showing that robust defense and low latency are achievable in edge private clouds. The findings offer actionable guidance for practitioners to deploy secure, cost-conscious LLM services in limited-resource environments while preserving tenant isolation and regulatory compliance.

Abstract

Large Language Model (LLM)-based question-answering systems offer significant potential for automating customer support and internal knowledge access in small businesses, yet their practical deployment remains challenging due to infrastructure costs, engineering complexity, and security risks, particularly in retrieval-augmented generation (RAG)-based settings. This paper presents an industry case study of an open-source, multi-tenant platform that enables small businesses to deploy customised LLM-based support chatbots via a no-code workflow. The platform is built on distributed, lightweight k3s clusters spanning heterogeneous, low-cost machines and interconnected through an encrypted overlay network, enabling cost-efficient resource pooling while enforcing container-based isolation and per-tenant data access controls. In addition, the platform integrates practical, platform-level defences against prompt injection attacks in RAG-based chatbots, translating insights from recent prompt injection research into deployable security mechanisms without requiring model retraining or enterprise-scale infrastructure. We evaluate the proposed platform through a real-world e-commerce deployment, demonstrating that secure and efficient LLM-based chatbot services can be achieved under realistic cost, operational, and security constraints faced by small businesses.

Securing LLM-as-a-Service for Small Businesses: An Industry Case Study of a Distributed Chatbot Deployment Platform

TL;DR

The paper tackles the challenge of deploying LLM-based support chatbots for small businesses by presenting an open-source, multi-tenant platform built on distributed, low-cost k3s clusters with an encrypted overlay. It introduces a layered security approach against prompt injection in RAG-based chatbots, combining prompt-level guard prompts with a model-agnostic GenTel-Shield detector, deployed without model retraining. Through a real-world e-commerce case study, the work evaluates both security effectiveness and deployment efficiency, showing that robust defense and low latency are achievable in edge private clouds. The findings offer actionable guidance for practitioners to deploy secure, cost-conscious LLM services in limited-resource environments while preserving tenant isolation and regulatory compliance.

Abstract

Large Language Model (LLM)-based question-answering systems offer significant potential for automating customer support and internal knowledge access in small businesses, yet their practical deployment remains challenging due to infrastructure costs, engineering complexity, and security risks, particularly in retrieval-augmented generation (RAG)-based settings. This paper presents an industry case study of an open-source, multi-tenant platform that enables small businesses to deploy customised LLM-based support chatbots via a no-code workflow. The platform is built on distributed, lightweight k3s clusters spanning heterogeneous, low-cost machines and interconnected through an encrypted overlay network, enabling cost-efficient resource pooling while enforcing container-based isolation and per-tenant data access controls. In addition, the platform integrates practical, platform-level defences against prompt injection attacks in RAG-based chatbots, translating insights from recent prompt injection research into deployable security mechanisms without requiring model retraining or enterprise-scale infrastructure. We evaluate the proposed platform through a real-world e-commerce deployment, demonstrating that secure and efficient LLM-based chatbot services can be achieved under realistic cost, operational, and security constraints faced by small businesses.
Paper Structure (22 sections, 2 figures, 2 tables)

This paper contains 22 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Deployment Context
  3. Target Users and Usage Scenarios
  4. Operational and Infrastructure Constraints
  5. Regulatory Requirements
  6. Platform Architecture
  7. Secure Design of RAG-based LLM Platforms
  8. Threat Model
  9. Mitigation Strategy
  10. Prompt-level Guard Prompts
  11. Prompt Injection Detection with GenTel-Shield
  12. Case Study: A Secure Multi-Tenant RAG-Based E-Commerce Chatbot Platform
  13. Deployment Setup
  14. Knowledge Base Preparation
  15. Prompt Injection Defence Configuration
  16. ...and 7 more sections

Figures (2)

  • Figure 1: The platform operates on a lightweight, Kubernetes-based edge private cloud interconnected via a secure overlay network. User requests are routed through a load-balancing layer to AI-powered chatbot services deployed on the cluster. The Kubernetes control plane runs on commodity machines, while inference workloads execute on DGX Spark--based accelerator nodes, providing large shared memory for efficient and cost-effective LLM inference without dedicated GPUs. The control plane manages scheduling and fault recovery across heterogeneous resources, enabling dynamic workload placement and resilient, secure AI service hosting.
  • Figure 2: Security-aware RAG workflow with layered prompt injection defences. Tenant documents are pre-processed through PII screening and de-identification before indexing. At runtime, user queries and retrieved context are filtered by GenTel-Shield and constrained by system-level guard prompts prior to LLM generation.