Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Research

TL;DR

This study demonstrates that small, fine-tuned open-source LLMs can achieve equal or superior performance to models such as ChatGPT-4, and proposes a hybrid workflow that leverages the strengths of both open and closed models, offering a balanced approach to performance, transparency, and reproducibility.

Abstract

Large Language Models (LLMs) are distinguished by their architecture, which dictates their parameter size and performance capabilities. Social scientists have increasingly adopted LLMs for text classification tasks, which are difficult to scale with human coders. While very large, closed-source models often deliver superior performance, their use presents significant risks. These include lack of transparency, potential exposure of sensitive data, challenges to replicability, and dependence on proprietary systems. Additionally, their high costs make them impractical for large-scale research projects. In contrast, open-source models, although available in various sizes, may underperform compared to commercial alternatives if used without further fine-tuning. However, open-source models offer distinct advantages: they can be run locally (ensuring data privacy), fine-tuned for specific tasks, shared within the research community, and integrated into reproducible workflows. This study demonstrates that small, fine-tuned open-source LLMs can achieve equal or superior performance to models such as ChatGPT-4. We further explore the relationship between training set size and fine-tuning efficacy in open-source models. Finally, we propose a hybrid workflow that leverages the strengths of both open and closed models, offering a balanced approach to performance, transparency, and reproducibility.

Figures (5)

• Figure 1: Acceleration of labeled datasets creation for fine-tuning using multiple LLMs and human verification.
• Figure 2: Accuracy for different LLAMA models and ChatGPT4 on both the training and the test sets. The 'FT' suffix stands for 'fine-tuned version' of the same model. Number of model parameters in the log-scale. For ChatGPT the real dimension of the model is not known, but seems to be built on top of 8 models of about 220B parameters.
• Figure 3: Hamming loss for different LLAMA models and ChatGPT4 on both the training and the test sets. The 'FT' suffix stands for 'fine-tuned version' of the same model. Number of model parameters in the log-scale. For ChatGPT the real dimension of the model is not known, but seems to be built on top of 8 models of about 220B parameters.
• Figure 4: Jaccard Index for different LLAMA models and ChatGPT4 on both the training and the test sets. The 'FT' suffix stands for 'fine-tuned version' of the same model. Number of model parameters in the log-scale. For ChatGPT the real dimension of the model is not known, but seems to be built on top of 8 models of about 220B parameters.
• Figure 5: Balanced accuracy, F1 score, Sensitivity and Specificity by major policy areas of the Comparative Agenda Project, for the classification of the both the train and test set using the LLAMA-2 fine-tuned model and on the verified set classified by ChatGPT4.