NeurIPS 2023 LLM Efficiency Fine-tuning Competition
Mark Saroufim, Yotam Perlitz, Leshem Choshen, Luca Antiga, Greg Bowyer, Christian Puhrsch, Driss Guessous, Supriya Rao, Geeta Chauhan, Ashvini Kumar, Jindal Pawan Kumar, Rajpoot Ankur Parikh, Joe Isaacson, Weiwei Yang
TL;DR
The NeurIPS 2023 LLM Efficiency Challenge investigates how to fine-tune open-source LLMs within 24 hours on a single GPU to democratize access. Using two tracks and an open/closed HELM-based evaluation with time limits, the study reveals that benchmark-based scoring can overfit and that data curation is crucial, as captured by the scoring rule $score = \left(\prod_{i=1}^{n} \text{mean\_win\_rate}(\text{scenario}_i)\right)^{1/n}$ and $FinalScore = \frac{1}{3}\text{OpenEvalScore} + \frac{2}{3}\text{ClosedEvalScore}$. Findings show winners relied on data-centric tuning with open-source libraries (e.g., Qwen-14B, Mistral-7B) and, in some cases, data-driven quantization or rank-decomposition approaches such as LLaMa-Factory, rather than novel architectures. The paper emphasizes reproducibility, transparency of artifacts, and the necessity of robust, real-world evaluation for meaningful progress in LLM fine-tuning.
Abstract
Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark datasets, mirroring the broader issue of benchmark overfitting on popular leaderboards and that data curation is essential in order to get a high performing LLM. The competition, which consisted of two stages - an open evaluation stage with publicly available tasks and a closed evaluation stage with unseen tasks - allowed us to assess the generalizability of fine-tuned LLMs. Our results highlight the limitations of current benchmark-based evaluation schemes for generative models and demonstrate the need for more robust evaluation methods. Notably, the winning submissions utilized standard open-source libraries and focused primarily on data curation. To facilitate further research and promote reproducibility, we release all competition entries, Docker files, and evaluation infrastructure, providing a valuable resource for the community to explore fine-tuning, overfitting, and reproducibility in LLMs.