DEP: A Decentralized Large Language Model Evaluation Protocol

TL;DR

A Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks, that enables modular, plug-and-play evaluation and reduces the cost of deploying benchmark evaluations.

Abstract

With the rapid development of Large Language Models (LLMs), a large number of benchmarks have been proposed. However, most benchmarks lack unified evaluation standard and require the manual implementation of custom scripts, making results hard to ensure consistency and reproducibility. Furthermore, mainstream evaluation frameworks are centralized, with datasets and answers, which increases the risk of benchmark leakage. To address these issues, we propose a Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks. The server can be mounted locally or deployed remotely, and once adapted, it can be reused over the long term. By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation: benchmark files and evaluation logic stay exclusively on the server side. In remote setting, users cannot access the ground truth, thereby achieving data isolation and leak-proof evaluation. To facilitate practical adoption, we develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control. We also provide detailed documentation for adapting new benchmarks to DEP. Using DEP toolkit, we evaluate multiple LLMs across benchmarks. Experimental results verify the effectiveness of DEP and show that it reduces the cost of deploying benchmark evaluations. As of February 2026, we have adapted over 60 benchmarks and continue to promote community co-construction to support unified evaluation across various tasks and domains.

Figures (7)

• Figure 1: Comparison between centralized framework and our DEP. Red denotes benchmark files while blue evaluation modules.
• Figure 2: Overview of DEP. The client connects to the LLMs and benchmarks via a unified interface. The LLM Adapter standardizes various LLM interfaces and inference methods, while the benchmark server handles requests from the client, such as data loading and evaluation submission.
• Figure 3: The screenshot of DEP website.
• Figure 4: Main results in the LLM evaluation with DEP.
• Figure 5: Server-side interface for remote evaluation.