TVS Sidekick: Challenges and Practical Insights from Deploying Large Language Models in the Enterprise

TL;DR

The paper investigates deploying large language models (LLMs) in an enterprise through TVS Sidekick, an in-house AI assistant that uses Retrieval Augmented Generation (RAG) to answer questions against internal data via a Microsoft Teams extension. The authors detail a two-pipeline architecture: an ingestion pipeline that vectorises company documents into a vector database and a RAG pipeline that routes user queries through retrieval, augmentation, and generation steps, including code-aware processing. Key contributions include (i) technical innovations in prompt engineering and RAG for enterprise data, (ii) regulatory alignment with the EU Artificial Intelligence Act (EU AIA) and pursuit of ISO/IEC 42001 as a harmonised standard, and (iii) sociotechnical insights from adoption metrics and qualitative user feedback guiding continual improvement. The study demonstrates practical governance and regulatory considerations for enterprise AI deployment and offers actionable guidance on monitoring, evaluation, and risk management in real-world logistics contexts.

Abstract

Many enterprises are increasingly adopting Artificial Intelligence (AI) to make internal processes more competitive and efficient. In response to public concern and new regulations for the ethical and responsible use of AI, implementing AI governance frameworks could help to integrate AI within organisations and mitigate associated risks. However, the rapid technological advances and lack of shared ethical AI infrastructures creates barriers to their practical adoption in businesses. This paper presents a real-world AI application at TVS Supply Chain Solutions, reporting on the experience developing an AI assistant underpinned by large language models and the ethical, regulatory, and sociotechnical challenges in deployment for enterprise use.

Figures (3)

• Figure 1: Overview of TVS Sidekick, an AI assistant that leverages LLMs to answer queries with relevant enterprise data using retrieval augmented generation (RAG) via a Microsoft Teams extension.
• Figure 2: Architecture diagram showing the main components of Sidekick, namely the ingestion and RAG pipelines, with novel approaches to prompt engineering (to handle code queries) and augmentation retrieval (for tool use).
• Figure 3: Monitoring system measuring real-time usage data of TVS Sidekick.