Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System
Geonwoo Bang, Dongho Kim, Moohong Min
TL;DR
This paper tackles the problem of converting user-defined criteria into explainable indicators for text quality in real-time settings. It introduces UniScore, a framework that uses lightweight per-criterion scoring by LLMs, measures discriminative power with Jensen–Shannon distance, and derives interpretable, signed weights via a difference-mapped Analytic Hierarchy Process, enabling fast, transparent aggregation. Experiments across Amazon Reviews, RoSE XSum, and Depression Tweet demonstrate that UniScore achieves strong predictive power and discriminative performance while maintaining interpretability and low latency, often outperforming or matching flagship evaluators. The work offers a practically impactful approach for live web services requiring explainable, multi-criteria text evaluation.
Abstract
Evaluating complex texts across domains requires converting user defined criteria into quantitative, explainable indicators, which is a persistent challenge in search and recommendation systems. Single prompt LLM evaluations suffer from complexity and latency issues, while criterion specific decomposition approaches rely on naive averaging or opaque black-box aggregation methods. We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process. Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensen Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices. Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.