InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation

TL;DR

The paper addresses the unreliable and opaque nature of centralized LLM evaluation by introducing InfiCoEvalChain, a blockchain-based decentralized framework (CoEvalChain) that aggregates diverse hardware, inference configurations, and independent validators to stabilize performance estimates. It presents a two-layer architecture (collaborative evaluation layer and blockchain foundation) and a probabilistic evaluation framework that uses a Schelling Point–driven consensus and a Commit-Reveal protocol to ensure fairness and integrity, along with a median-based, MAD-informed incentive mechanism. Empirical results show a substantial reduction in evaluation variance, lowering the standard deviation from $1.67$ to $0.28$ across benchmarks, with tighter 95% confidence intervals and robust performance across model scales and hard tasks. The work promises a more trustworthy, scalable, and globally participatory evaluation ecosystem that can guide model optimization and mitigate issues such as data leakage and overfitting in benchmarks.

Abstract

The rapid advancement of large language models (LLMs) demands increasingly reliable evaluation, yet current centralized evaluation suffers from opacity, overfitting, and hardware-induced variance. Our empirical analysis reveals an alarming inconsistency in existing evaluations: the standard deviation across ten repeated runs of a single model on HumanEval (1.67) actually exceeds the performance gap among the top-10 models on the official leaderboard (0.91), rendering current rankings statistically precarious. To mitigate these instabilities, we propose a decentralized evaluation framework that enables hardware and parameter diversity through large-scale benchmarking across heterogeneous compute nodes. By leveraging the blockchain-based protocol, the framework incentivizes global contributors to act as independent validators, using a robust reward system to ensure evaluation integrity and discourage dishonest participation. This collective verification transforms evaluation from a "centralized black box" into a "decentralized endorsement" where multi-party consensus and diverse inference environments yield a more stable, representative metric. Experimental results demonstrate that the decentralized evaluation framework reduces the standard deviation across ten runs on the same model to 0.28. This significant improvement over conventional frameworks ensures higher statistical confidence in model rankings. We have completely implemented this platform and will soon release it to the community.

Figures (4)

• Figure 1: Performance fluctuations of various LLMs Each model was evaluated 10 times on a single computing node with minor variations in inference parameters. The observed variance across all benchmarks highlights the inherent instability of model performance in traditional centralized evaluation settings.
• Figure 2: Overview of the CoEvalChain. The framework coordinates heterogeneous computing nodes for collaborative AI tasks (left). An algorithmic incentive mechanism distributes rewards or penalties based on evaluation metrics (top right), while a blockchain-based P2P network ensures integrity through a multi-stage consensus protocol (bottom right).
• Figure 3: Stability Comparison of Evaluation Paradigms on Large-scale Closed-source Models. Across multiple benchmarks (GPQA-Diamond, GSM8K, and Humaneval), the decentralized evaluation framework (dashed lines) consistently exhibits significantly lower variance compared to the centralized approach (solid lines). This trend holds true even for high-parameter models such as DeepSeek-V3.2 and Gemini-2.5-Flash, demonstrating that decentralization effectively mitigates the inherent stochasticity in large-scale model inference.
• Figure 4: Distribution of Evaluator Incentivization Across the Decentralized Experiment. This histogram illustrates the distribution of incentives allocated to evaluators across all benchmark sessions within our framework. The data exhibits a clear Gaussian-like distribution centered around the collective consensus, demonstrating the consistency and fairness of the incentive mechanism. Notably, the presence of isolated frequency bars near zero represents the effective identification and penalization of outliers—evaluators whose submissions deviated significantly from the robust median baseline.