Large Language Model Evaluation Via Multi AI Agents: Preliminary results
Zeeshan Rasheed, Muhammad Waseem, Kari Systä, Pekka Abrahamsson
TL;DR
The paper presents a novel multi-agent evaluation framework that uses eight distinct AI agents, each interfacing with a different LLM to generate code from a common description, with a dedicated verification agent employing the HumanEval benchmark and the $pass@k$ metric to assess correctness and efficiency. It establishes an API-based, cross-model comparison of code-generation capabilities across GPT-4, GPT-4 Turbo, GPT-3.5, GPT-3.5 Turbo, Google Bard, LLAMA, and Hugging Face, reporting that GPT-3.5 Turbo leads on a 10-prompt set. The study lays the groundwork for a more robust evaluation by planning MBPP integration, expanding prompts, and engaging twenty practitioners for feedback to refine the framework. By bridging multi-agent retrieval with automated verification, the work offers a scalable approach to benchmarking LLMs for code generation that can inform practice and further research.
Abstract
As Large Language Models (LLMs) have become integral to both research and daily operations, rigorous evaluation is crucial. This assessment is important not only for individual tasks but also for understanding their societal impact and potential risks. Despite extensive efforts to examine LLMs from various perspectives, there is a noticeable lack of multi-agent AI models specifically designed to evaluate the performance of different LLMs. To address this gap, we introduce a novel multi-agent AI model that aims to assess and compare the performance of various LLMs. Our model consists of eight distinct AI agents, each responsible for retrieving code based on a common description from different advanced language models, including GPT-3.5, GPT-3.5 Turbo, GPT-4, GPT-4 Turbo, Google Bard, LLAMA, and Hugging Face. Our developed model utilizes the API of each language model to retrieve code for a given high-level description. Additionally, we developed a verification agent, tasked with the critical role of evaluating the code generated by its counterparts. We integrate the HumanEval benchmark into our verification agent to assess the generated code's performance, providing insights into their respective capabilities and efficiencies. Our initial results indicate that the GPT-3.5 Turbo model's performance is comparatively better than the other models. This preliminary analysis serves as a benchmark, comparing their performances side by side. Our future goal is to enhance the evaluation process by incorporating the Massively Multitask Benchmark for Python (MBPP) benchmark, which is expected to further refine our assessment. Additionally, we plan to share our developed model with twenty practitioners from various backgrounds to test our model and collect their feedback for further improvement.