The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegram

TL;DR

The paper tackles the challenge of assessing credibility for URL-sparse Telegram messages by introducing Tag2Cred, a URL-agnostic framework that converts text into a closed vocabulary of rhetorical tags (Theme, Claim type, CTA, Evidence) and learns a calibrated risk score via $\ell_2$-regularized logistic regression after LLM-based tag assignment. It builds two supervision streams—MBFC-based distant supervision and a human-annotated gold tag codebook—facilitated by a fine-tuned Qwen-32B tagger and URL masking to prevent source leakage. TAG2CRED achieves strong discrimination and calibration under domain-disjoint evaluation (ROC-AUC $0.871$, macro-F1 $0.787$, Brier $0.167$), and an ensemble with TF-IDF and SBERT reaches ROC-AUC $0.901$, macro-F1 $0.813$, and ECE $0.084$, illustrating complementary information from style and content. Beyond per-message scoring, the paper demonstrates a monitoring workflow that analyzes risk-weighted tag dynamics, prototypes, and strategy families to reveal high-risk narratives, their concentration, and temporal drift, enabling scalable and interpretable interventions on short-text, URL-sparse platforms.

Abstract

Telegram has become one of the leading platforms for disseminating misinformational messages. However, many existing pipelines still classify each message's credibility based on the reputation of its associated domain names or its lexical features. Such methods work well on traditional long-form news articles published by well-known sources, but high-risk posts on Telegram are short and URL-sparse, leading to failures for link-based and standard TF-IDF models. To this end, we propose the TAG2CRED pipeline, a method designed for such short, convoluted messages. Our model will directly score each post based on the tags assigned to the text. We designed a concise label system that covers the dimensions of theme, claim type, call to action, and evidence. The fine-tuned large language model (LLM) assigns tags to messages and then maps these tags to calibrated risk scores in the [0,1] interval through L2-regularized logistic regression. We evaluated 87,936 Telegram messages associated with Media Bias/Fact Check (MBFC), using URL masking and domain disjoint splits. The results showed that the ROC-AUC of the TAG2CRED model reached 0.871, the macro-F1 value was 0.787, and the Brier score was 0.167, outperforming the baseline TF-IDF (macro-F1 value 0.737, Brier score 0.248); at the same time, the number of features used in this model is much smaller, and the generalization ability on infrequent domains is stronger. The performance of the stacked ensemble model (TF-IDF + TAG2CRED + SBERT) was further improved over the baseline SBERT. ROC-AUC reached 0.901, and the macro-F1 value was 0.813 (Brier score 0.114). This indicates that style labels and lexical features may capture different but complementary dimensions of information risk.

Figures (4)

• Figure 1: Tag2Cred Pipeline.
• Figure 2: URL Sparsity in the Mega Corpus. Most messages contain no explicit web URL; among URL-bearing messages, single URL ones dominates.
• Figure 3: MBFC Credibility and Factuality Distributions for Rated URLs.
• Figure 4: Burstiness of selected tags (higher indicates sharper short-term spikes).