Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Porting an LLM based Application from ChatGPT to an On-Premise Environment

Teemu Paloniemi, Manu Setälä, Tommi Mikkonen

TL;DR

The study investigates porting an LLM-based procurement assistant (AIPA) from ChatGPT to an on-premise environment to address privacy, security, customization, and cost concerns under EU regulations. It adopts a case study and Design Science Research to outline a three-step porting process (preparation, implementation, deployment/evaluation), including code changes, hardware choices, and model selection; it implements a local API via llama.cpp and applies LoRA-based fine-tuning, with server-side operation to mitigate data leakage. The findings indicate that on-prem porting is feasible, though performance may differ from the cloud baseline and may require additional training or larger models; the approach offers improved data localization and cost control, enabling industry adoption. Limitations include the single-case scope and lack of extensive long-term evaluation, prompting call for broader porting studies, concept-drift analysis, and API-level interoperability to replace LLMs with minimal code changes.

Abstract

Given the data-intensive nature of Machine Learning (ML) systems in general, and Large Language Models (LLM) in particular, using them in cloud based environments can become a challenge due to legislation related to privacy and security of data. Taking such aspects into consideration implies porting the LLMs to an on-premise environment, where privacy and security can be controlled. In this paper, we study this porting process of a real-life application using ChatGPT, which runs in a public cloud, to an on-premise environment. The application being ported is AIPA, a system that leverages Large Language Models (LLMs) and sophisticated data analytics to enhance the assessment of procurement call bids. The main considerations in the porting process include transparency of open source models and cost of hardware, which are central design choices of the on-premise environment. In addition to presenting the porting process, we evaluate downsides and benefits associated with porting.

Porting an LLM based Application from ChatGPT to an On-Premise Environment

TL;DR

The study investigates porting an LLM-based procurement assistant (AIPA) from ChatGPT to an on-premise environment to address privacy, security, customization, and cost concerns under EU regulations. It adopts a case study and Design Science Research to outline a three-step porting process (preparation, implementation, deployment/evaluation), including code changes, hardware choices, and model selection; it implements a local API via llama.cpp and applies LoRA-based fine-tuning, with server-side operation to mitigate data leakage. The findings indicate that on-prem porting is feasible, though performance may differ from the cloud baseline and may require additional training or larger models; the approach offers improved data localization and cost control, enabling industry adoption. Limitations include the single-case scope and lack of extensive long-term evaluation, prompting call for broader porting studies, concept-drift analysis, and API-level interoperability to replace LLMs with minimal code changes.

Abstract

Given the data-intensive nature of Machine Learning (ML) systems in general, and Large Language Models (LLM) in particular, using them in cloud based environments can become a challenge due to legislation related to privacy and security of data. Taking such aspects into consideration implies porting the LLMs to an on-premise environment, where privacy and security can be controlled. In this paper, we study this porting process of a real-life application using ChatGPT, which runs in a public cloud, to an on-premise environment. The application being ported is AIPA, a system that leverages Large Language Models (LLMs) and sophisticated data analytics to enhance the assessment of procurement call bids. The main considerations in the porting process include transparency of open source models and cost of hardware, which are central design choices of the on-premise environment. In addition to presenting the porting process, we evaluate downsides and benefits associated with porting.

Paper Structure

This paper contains 19 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Large Language Models
  4. Public versus On-Premise Cloud Services
  5. Porting ML Models
  6. Research Approach
  7. Research Context
  8. Research Questions
  9. Research Methods
  10. Design and Implementation
  11. Porting Strategy
  12. Preparation
  13. Code changes
  14. Hardware Specifications
  15. Model Selection
  16. ...and 4 more sections

Figures (3)

  • Figure 1: AIPA high-level architecture. Running in the cloud, AI assisted search function is the key feature of the system.
  • Figure 2: AIPA in action, with subsystem interaction visualized.
  • Figure 3: Steps of the porting process.