Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models

TL;DR

This work tackles access and privacy barriers of proprietary LLMs by introducing domain-agnostic self-refinement, which improves open-source models without external feedback, and PeRFICS, a cost-aware ranking metric. Empirical results show an average improvement of 8.2% across 7B–65B models, with notable gains for Vicuna-7B and Vicuna-13B, the latter even outperforming ChatGPT after refinement. The combination of a domain-agnostic refinement process and PeRFICS enables informed model selection under resource and privacy constraints, demonstrated through case studies in email automation, game NPCs, and corporate coding tasks. Overall, the approach advances democratization of LLMs by enabling high performance on consumer hardware while reducing reliance on proprietary systems.

Abstract

The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. High-performing open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance. To address this gap, we propose (1) A untargeted variant of iterative self-critique and self-refinement devoid of external influence. (2) A novel ranking metric - Performance, Refinement, and Inference Cost Score (PeRFICS) - to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement. This work has profound implications for resource-constrained and information-sensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy. The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by case studies.

Figures (4)

• Figure 1: ChatGPT vs GPT-4X-Alpasta baseline vs GPT-4X-Alpasta refined
• Figure 2: % Change in token generation post-refinement.
• Figure 3: % Wins against ChatGPT for zero-shot responses
• Figure 4: % Wins against ChatGPT for refined responses