TruthTensor: Evaluating LLMs Human Imitation through Prediction Market Drift and Holistic Reasoning

TL;DR

TruthTensor introduces a dynamic evaluation paradigm that anchors LLM assessment to live prediction markets, focusing on human imitation rather than static accuracy. It formalizes drift as $D = D_n + D_t + D_c$ and calibrates forecasts against market-implied probabilities using metrics like the Brier score and $ECE$/$MCE$ to capture calibration and narrative stability. The framework employs an Instruction Locking mechanism and four core algorithms—Drift Measurement, Baseline Comparison, Holistic Human Imitation Score ($HHIS$), and Risk Assessment—to produce a Holistic Human Imitation Score $H = 0.2C + 0.2Cal + 0.3(1 - D) + 0.15R + 0.15Q$. In a live benchmark across hundreds of markets and eight frontier-scale models, TruthTensor shows that similar forecast accuracy can diverge in calibration, narrative stability, and risk sensitivity, underscoring the need for multi-axis evaluation and open, versioned evaluation contracts. The approach offers a scalable, reproducible, and economically grounded framework for evaluating LLMs in real-world decision contexts.

Abstract

Evaluating language models and AI agents remains fundamentally challenging because static benchmarks fail to capture real-world uncertainty, distribution shift, and the gap between isolated task accuracy and human-aligned decision-making under evolving conditions. This paper introduces TruthTensor, a novel, reproducible evaluation paradigm that measures Large Language Models (LLMs) not only as prediction engines but as human-imitation systems operating in socially-grounded, high-entropy environments. Building on forward-looking, contamination-free tasks, our framework anchors evaluation to live prediction markets and combines probabilistic scoring to provide a holistic view of model behavior. TruthTensor complements traditional correctness metrics with drift-centric diagnostics and explicit robustness checks for reproducibility. It specify human vs. automated evaluation roles, annotation protocols, and statistical testing procedures to ensure interpretability and replicability of results. In experiments across 500+ real markets (political, economic, cultural, technological), TruthTensor demonstrates that models with similar forecast accuracy can diverge markedly in calibration, drift, and risk-sensitivity, underscoring the need to evaluate models along multiple axes (accuracy, calibration, narrative stability, cost, and resource efficiency). TruthTensor therefore operationalizes modern evaluation best practices, clear hypothesis framing, careful metric selection, transparent compute/cost reporting, human-in-the-loop validation, and open, versioned evaluation contracts, to produce defensible assessments of LLMs in real-world decision contexts. We publicly release TruthTensor at https://truthtensor.com

Figures (6)

• Figure 1: TruthTensor System Architecture.
• Figure 2: Adjusted cumulative P&L versus historical baseline (left) and realized cumulative trading P&L (right).
• Figure 3: Distribution of strategy selection frequency by model.
• Figure 4: Average input and output token consumption by model.
• Figure 5: Depth of model adjustments across edge, expected return, and probability dimensions (log scale).