Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

Nicholas Santavas, Kareem Eissa, Patrycja Cieplicka, Piotr Florek, Matteo Nulli, Stefan Vasilev, Seyyed Hadi Hashemi, Antonios Gasteratos, Shahram Khadivi

TL;DR

OptiKIT introduces an end-to-end, distributed framework for automated enterprise LLM optimization that orchestrates model compression, calibration, statistical evaluation, benchmarking, and deployment tuning. By employing backend-agnostic compression, recipe-driven quantization, SLO-driven benchmarking, and Bayesian runtime tuning within a Ray-based architecture, it achieves substantial throughput gains per GPU while maintaining near full-precision accuracy across diverse tasks. The work demonstrates robust production-ready automation, reproducibility, and enterprise integration, reducing manual optimization time and enabling non-expert teams to deploy optimized LLMs at scale. Its practical impact lies in democratizing high-performance model deployment under constrained compute resources, with open-source goals to encourage external contributions and reproducibility.

Abstract

Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset. This challenge is particularly evident in managing GPU utilization across heterogeneous infrastructure while enabling teams with diverse workloads and limited LLM optimization experience to deploy models efficiently. We present OptiKIT, a distributed LLM optimization framework that democratizes model compression and tuning by automating complex optimization workflows for non-expert teams. OptiKIT provides dynamic resource allocation, staged pipeline execution with automatic cleanup, and seamless enterprise integration. In production, it delivers more than 2x GPU throughput improvement while empowering application teams to achieve consistent performance improvements without deep LLM optimization expertise. We share both the platform design and key engineering insights into resource allocation algorithms, pipeline orchestration, and integration patterns that enable large-scale, production-grade democratization of model optimization. Finally, we open-source the system to enable external contributions and broader reproducibility.

Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

TL;DR

OptiKIT introduces an end-to-end, distributed framework for automated enterprise LLM optimization that orchestrates model compression, calibration, statistical evaluation, benchmarking, and deployment tuning. By employing backend-agnostic compression, recipe-driven quantization, SLO-driven benchmarking, and Bayesian runtime tuning within a Ray-based architecture, it achieves substantial throughput gains per GPU while maintaining near full-precision accuracy across diverse tasks. The work demonstrates robust production-ready automation, reproducibility, and enterprise integration, reducing manual optimization time and enabling non-expert teams to deploy optimized LLMs at scale. Its practical impact lies in democratizing high-performance model deployment under constrained compute resources, with open-source goals to encourage external contributions and reproducibility.

Abstract

Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset. This challenge is particularly evident in managing GPU utilization across heterogeneous infrastructure while enabling teams with diverse workloads and limited LLM optimization experience to deploy models efficiently. We present OptiKIT, a distributed LLM optimization framework that democratizes model compression and tuning by automating complex optimization workflows for non-expert teams. OptiKIT provides dynamic resource allocation, staged pipeline execution with automatic cleanup, and seamless enterprise integration. In production, it delivers more than 2x GPU throughput improvement while empowering application teams to achieve consistent performance improvements without deep LLM optimization expertise. We share both the platform design and key engineering insights into resource allocation algorithms, pipeline orchestration, and integration patterns that enable large-scale, production-grade democratization of model optimization. Finally, we open-source the system to enable external contributions and broader reproducibility.
Paper Structure (50 sections, 2 equations, 5 figures, 7 tables, 2 algorithms)

This paper contains 50 sections, 2 equations, 5 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The LLM Optimization Challenge at Scale
  3. System Design and Architecture
  4. Design Philosophy
  5. Motivation
  6. Abstractions
  7. Architecture Components
  8. Actor-Based Execution Layer
  9. Flow Composition Layer
  10. Submission Engine Layer
  11. Core Subsystems
  12. Optimizer: Universal Compression Framework
  13. Backend-Agnostic Architecture
  14. Recipe-Based Configuration System
  15. Calibration Data Sampling
  16. ...and 35 more sections

Figures (5)

  • Figure 1: OptiKIT time and throughput gains. The top figure shows the engineering time saved in model optimization through OptiKIT vs human hours. In the bottom figure the optimal TPS (Transactions Per Second i.e., Throughput) after the OptiKIT cycle has terminated vs the baseline TPS. We report results on three model families. Human hours are estimated on internal data.
  • Figure 2: OptiKIT full pipeline. The figure shows the full OptiKIT flow. We begin by fetching any base/instruct model along with calibration data if needed, and apply model compression through the user selected technique. We then proceed to perform a statistical evaluation of the optimized model to ensure the validity of our compression strategy. If the performance is up to standards, we determine the set of parameter space for deployment tuning. Subsequently, we sample from this space and perform Inference Benchmarking to determine the optimal sub-set of parameters for deployment. If the SLOs (Service Level Objectives) is not met we iteratively repeat the above, sampling a new set of parameters. When a parameter configuration meets the SLOs, we return the model configuration along with its weights, ready for deployment. The small logos represent part of the back-ends supported.
  • Figure 3: OptiKIT system architecture. The figure illustrates the modular orchestration of distributed LLM optimization workflows. The central orchestration layer manages workflow submission, resource allocation, and experiment tracking via Ray Actors, integrating with external data and model sources (HDFS, MMS, EMS) and underlying heterogeneous Ray clusters (H200, H100, A100 nodes). Supporting libraries for compression, benchmarking, and statistical evaluation provide extensible optimization capabilities, while monitoring and telemetry ensure observability through Grafana, logs, and tracing.
  • Figure 4: Regression diagnostics for a Benchmarker trial. A fitted slope $\beta \approx 1$ (green) indicates steady-state operation queuing while $\beta > 1$ (red) denotes an overload regime.
  • Figure 5: Total OptiKIT Runtime per Model. We show for each model family the total optimization flow time. Mistral Small 3 24B has two usage scenarios, respectively with 3k/0.2k and 1.5k/1.5k input/output sizes.