From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs

TL;DR

This work tackles the cost–performance bottleneck in deploying large language models by proposing a three-stage end-to-end pipeline—prototyping, knowledge transfer, and model compression—that yields super-tiny LLMs with competitive or superior domain performance. It introduces a novel hybrid knowledge transfer approach that combines reinforcement learning with knowledge distillation to preserve and transfer reasoning capabilities from a large teacher to 0.5B models, followed by further compression to 0.4B. The methodology is validated with extensive experiments showing substantial inference speedups ($e.g.,$ up to $10\times$–$100\times$) and significant cost reductions while maintaining high accuracy, including cross-domain effectiveness and practical deployment on commodity GPUs. The work also introduces LLM-as-a-Judge for data-quality assessment, supports cross-domain applicability, and provides a roadmap for scalable, cost-efficient LLM deployment beyond the AGI horizon.

Abstract

Large Language Models (LLMs) have significantly advanced artificial intelligence by optimizing traditional Natural Language Processing (NLP) workflows, facilitating their integration into various systems. Many such NLP systems, including ours, directly incorporate LLMs. However, this approach either results in expensive costs or yields suboptimal performance after fine-tuning. In this paper, we introduce a three-stage cost-efficient end-to-end LLM deployment pipeline, comprising prototyping, knowledge transfer, and model compression, to effectively tackle the cost-performance dilemma in LLM-based frameworks. Its high cost-efficiency is manifested not only in simplifying system complexity and producing super-tiny online models with enhanced performance and reduced costs in the results, but also in addressing development cycle constraints, the lack of extensive high-quality data, and limited computational resources during the project development process. In the first stage, we construct an optimal performance prototype system by transforming complex tasks into a function call-based LLM-driven pipeline, which serves as a teacher model to generate high-quality data. In the second stage, we combine techniques like rejection sampling fine-tuning, reinforcement learning, and knowledge distillation to transfer knowledge to 0.5B student models, delivering effective performance at minimal cost. In the final stage, we further compress models to 0.4B via quantization and pruning, achieving ultra-low latency and cost. Extensive experimental results and the framework's modular design suggest cross-domain capabilities and potential applicability in other NLP areas.

Figures (9)

• Figure 1: Workflows of prototyping, knowledge transfer and compression.
• Figure 2: The workflow of the hybrid knowledge transfer strategy.
• Figure 3: Cost and performance results. FLOPs reduced by 10x with nearly lossless AR.
• Figure 4: Comparison of annotation results between the LLM prototype and the BERT-based system determined by professional annotators.
• Figure 5: Comparison of RT metrics using different quantization and pruning strategies.