LLMChain: Blockchain-based Reputation System for Sharing and Evaluating Large Language Models

TL;DR

LLMChain tackles the trust problem in LLMs by introducing a decentralized blockchain-based reputation system that fuses automatic evaluation with human feedback to produce context-aware reputations for LLMs. The framework deploys a four-layer architecture with off-chain IPFS storage and an oracle network, and implements a three-phase evaluation flow (Registration, Sharing, Evaluation) controlled by smart contracts. The reputation model combines $R^{a}$, $R^{h}$, and $R$ into a unified update mechanism that responds to both automatic and human signals, with dynamic weighting and a trust threshold to handle good and bad behavior. Empirical results on MTBench, GooAQ, and LLMGooAQ show strong alignment between automatic metrics (notably BARTScore) and human judgments, while a Hyperledger Besu deployment demonstrates practical throughput and scalability, supporting real-world adoption and model improvement via user and provider feedback.

Abstract

Large Language Models (LLMs) have witnessed rapid growth in emerging challenges and capabilities of language understanding, generation, and reasoning. Despite their remarkable performance in natural language processing-based applications, LLMs are susceptible to undesirable and erratic behaviors, including hallucinations, unreliable reasoning, and the generation of harmful content. These flawed behaviors undermine trust in LLMs and pose significant hurdles to their adoption in real-world applications, such as legal assistance and medical diagnosis, where precision, reliability, and ethical considerations are paramount. These could also lead to user dissatisfaction, which is currently inadequately assessed and captured. Therefore, to effectively and transparently assess users' satisfaction and trust in their interactions with LLMs, we design and develop LLMChain, a decentralized blockchain-based reputation system that combines automatic evaluation with human feedback to assign contextual reputation scores that accurately reflect LLM's behavior. LLMChain not only helps users and entities identify the most trustworthy LLM for their specific needs, but also provides LLM developers with valuable information to refine and improve their models. To our knowledge, this is the first time that a blockchain-based distributed framework for sharing and evaluating LLMs has been introduced. Implemented using emerging tools, LLMChain is evaluated across two benchmark datasets, showcasing its effectiveness and scalability in assessing seven different LLMs.

Figures (6)

• Figure 1: Overview of the LLMChain framework. \ref{['fig1a']} presents the layered BC-powered architecture. It consists of four main layers: a user layer formed by individuals with different expertise, a BC layer built on a consortium BC managed by LLM providers, and an Oracle layer built up by a decentralized network interconnecting the BC layer with LLMs layer. \ref{['fig1b']} describes the LLMs evaluation process in LLMChain.
• Figure 2: The Effectiveness of LLMChain's Reputation model under different $\mathcal{W}^h$ and $D$.
• Figure 3: -. Labels are denoted as: {"0:Llama-13B", "1:Alpaca-13B", "2:Vicuna-13B", "3:GPT-3.5", "4:Claud-v1"}
• Figure 5: Ground-Truth Answers vs Vicuna-13B Answers as References for BARTScore-based Pairwise-comparison on the LLMGooAQ dataset. Labels are denoted as: {0: "Alpaca-13b", 1: "Llama-2-13b", 2: "Chatglm-6b", 3: "Fastchat-t5-3b", 4: "Koala-13b", 5: "Vicuna-7b", 6: "Vicuna-13b"}.
• Figure 6: Changes in $R^{a}$, $R^{h}$, and $R$ of seven LLMs using LLMGooAQ.