TRACE: Transparent Web Reliability Assessment with Contextual Explanations
Joydeep Chandra, Aleksandr Algazinov, Satyam Kumar Navneet, Rim El Filali, Matt Laing, Andrew Hanna, Yong Zhang
TL;DR
TRACE tackles the need for nuanced credibility assessment by replacing binary fact-checks with a continuous reliability score in the range $[0.1, 1.0]$ and providing contextual explanations. It introduces TrueGL-1B, a 1B-parameter model fine-tuned on a novel dataset of over $140{,}000$ articles annotated with $35$ distinct scores via a Human-LLM co-creation and data-poisoning pipeline, enabling fine-grained reliability regression. The approach demonstrates strong regression performance (e.g., $R^2 \approx 0.78$, MAE around $0.12$ on held-out data) and coherent justification generation, outperforming un-finetuned baselines and traditional rule-based methods, with zero-shot generalization to external benchmarks. By coupling score precision with structured explanations, TRACE aims to empower users with interpretable, actionable assessments that can be deployed in web search and browsing tools, advancing transparent AI-assisted information consumption.
Abstract
In an era of AI-generated misinformation flooding the web, existing tools struggle to empower users with nuanced, transparent assessments of content credibility. They often default to binary (true/false) classifications without contextual justifications, leaving users vulnerable to disinformation. We address this gap by introducing TRACE: Transparent Reliability Assessment with Contextual Explanations, a unified framework that performs two key tasks: (1) it assigns a fine-grained, continuous reliability score (from 0.1 to 1.0) to web content, and (2) it generates a contextual explanation for its assessment. The core of TRACE is the TrueGL-1B model, fine-tuned on a novel, large-scale dataset of over 140,000 articles. This dataset's primary contribution is its annotation with 35 distinct continuous reliability scores, created using a Human-LLM co-creation and data poisoning paradigm. This method overcomes the limitations of binary-labeled datasets by populating the mid-ranges of reliability. In our evaluation, TrueGL-1B consistently outperforms other small-scale LLM baselines and rule-based approaches on key regression metrics, including MAE, RMSE, and R2. The model's high accuracy and interpretable justifications make trustworthy information more accessible. To foster future research, our code and model are made publicly available here: github.com/zade90/TrueGL.