CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency

TL;DR

CryptoBench introduces a live, expert-curated benchmark for evaluating LLM agents in the volatile cryptocurrency domain. It emphasizes time-sensitivity, adversarial information environments, and multi-source synthesis, using a four-quadrant task taxonomy and monthly updates. The study reveals a retrieval-prediction imbalance and that agentic framing can shift model rankings, indicating raw capability does not fully translate to expert performance. The benchmark's dynamic, real-world data integration and interaction with on-chain data platforms offers a realistic measure of crypto-analytic ability and points toward domain-specific predictive architectures.

Abstract

This paper introduces CryptoBench, the first expert-curated, dynamic benchmark designed to rigorously evaluate the real-world capabilities of Large Language Model (LLM) agents in the uniquely demanding and fast-paced cryptocurrency domain. Unlike general-purpose agent benchmarks for search and prediction, professional crypto analysis presents specific challenges: \emph{extreme time-sensitivity}, \emph{a highly adversarial information environment}, and the critical need to synthesize data from \emph{diverse, specialized sources}, such as on-chain intelligence platforms and real-time Decentralized Finance (DeFi) dashboards. CryptoBench thus serves as a much more challenging and valuable scenario for LLM agent assessment. To address these challenges, we constructed a live, dynamic benchmark featuring 50 questions per month, expertly designed by crypto-native professionals to mirror actual analyst workflows. These tasks are rigorously categorized within a four-quadrant system: Simple Retrieval, Complex Retrieval, Simple Prediction, and Complex Prediction. This granular categorization enables a precise assessment of an LLM agent's foundational data-gathering capabilities alongside its advanced analytical and forecasting skills. Our evaluation of ten LLMs, both directly and within an agentic framework, reveals a performance hierarchy and uncovers a failure mode. We observe a \textit{retrieval-prediction imbalance}, where many leading models, despite being proficient at data retrieval, demonstrate a pronounced weakness in tasks requiring predictive analysis. This highlights a problematic tendency for agents to appear factually grounded while lacking the deeper analytical capabilities to synthesize information.

Figures (8)

• Figure 1: Combined Evaluation Scores of LLM and SmolAgent Frameworks between October $12^{th}$ to November $11^{th}$. Red bars represent direct LLM evaluation, while blue bars represent performance within the SmolAgent framework.
• Figure 2: Statistics of the CryptoBench dataset between October $12^{th}$ to November $11^{th}$.
• Figure 3: The CryptoBench Four-Quadrant Task Classification System. Tasks are categorized along two axes: Complexity (Simple vs. Complex) and Cognitive Demand (Retrieval vs. Prediction), providing a granular view of agent capabilities.
• Figure 4: The CryptoBench Dataset Construction and Dynamic Update Pipeline. The top panel illustrates the rigorous multi-stage verification protocol for creating question templates. The bottom panel shows the monthly process for generating fresh, solvable questions from the template pool to ensure the benchmark's timeliness and relevance.
• Figure 5: Overall Performance Comparison between October $12^{th}$ to November $11^{th}$. (a) Direct LLM evaluation results, showing a wide performance spread. (b) SmolAgent evaluation results, highlighting the impact of an agentic framework on model performance.