Selecting the Right LLM for eGov Explanations

TL;DR

The paper addresses how to select the right LLM for generating trustworthy, explanation-like content in e-government, by adapting a quality-scale to compare fidelity and interpretability of LLM-generated explanations in a tax refund scenario. It employs a 4×3 controlled online experiment with four LLM types and three inquiry cases, measuring citizen-perceived explanation quality while accounting for trust and curiosity as covariates. The results show significant LLM-type effects on fidelity and interpretability, with covariates influencing perceptions, and demonstrate the potential—and current limitations—of predictive models to estimate human feedback from textual embeddings and user traits. The work proposes a practical path toward scalable LLM selection in public services, including data-sharing and enhancements such as RAG and multi-criteria decision analysis to balance accuracy, trust, and resource costs.

Abstract

The perceived quality of the explanations accompanying e-government services is key to gaining trust in these institutions, consequently amplifying further usage of these services. Recent advances in generative AI, and concretely in Large Language Models (LLMs) allow the automation of such content articulations, eliciting explanations' interpretability and fidelity, and more generally, adapting content to various audiences. However, selecting the right LLM type for this has become a non-trivial task for e-government service providers. In this work, we adapted a previously developed scale to assist with this selection, providing a systematic approach for the comparative analysis of the perceived quality of explanations generated by various LLMs. We further demonstrated its applicability through the tax-return process, using it as an exemplar use case that could benefit from employing an LLM to generate explanations about tax refund decisions. This was attained through a user study with 128 survey respondents who were asked to rate different versions of LLM-generated explanations about tax refund decisions, providing a methodological basis for selecting the most appropriate LLM. Recognizing the practical challenges of conducting such a survey, we also began exploring the automation of this process by attempting to replicate human feedback using a selection of cutting-edge predictive techniques.

Figures (7)

• Figure 1: Tax refund process.
• Figure 2: Prompt example for explanation generation
• Figure 3: Box-plot distributions per LLM model type
• Figure 4: Adjusted Means for Fidelity
• Figure 5: Adjusted Means for Interpretability