What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practices

TL;DR

This study provides a systematic framework to identify and quantify hard and soft censorship in LLM outputs related to political figures, differentiating explicit refusals from selective omissions. It introduces a dual evaluation pipeline and a multilingual, cross-regional approach, applying them to a diverse set of models across six UN languages. The findings reveal substantial, region- and language-dependent censorship patterns, with domestic-audience tailoring evident in several models, and underscore the need for ideological diversity and greater transparency in moderation policies. By releasing an ommission dataset and open methodology, the work enables reproducible assessment of LLM moderation practices and informs governance and policy discussions on AI transparency and fairness.

Abstract

Large Language Models (LLMs) are increasingly deployed as gateways to information, yet their content moderation practices remain underexplored. This work investigates the extent to which LLMs refuse to answer or omit information when prompted on political topics. To do so, we distinguish between hard censorship (i.e., generated refusals, error messages, or canned denial responses) and soft censorship (i.e., selective omission or downplaying of key elements), which we identify in LLMs' responses when asked to provide information on a broad range of political figures. Our analysis covers 14 state-of-the-art models from Western countries, China, and Russia, prompted in all six official United Nations (UN) languages. Our analysis suggests that although censorship is observed across the board, it is predominantly tailored to an LLM provider's domestic audience and typically manifests as either hard censorship or soft censorship (though rarely both concurrently). These findings underscore the need for ideological and geographic diversity among publicly available LLMs, and greater transparency in LLM moderation strategies to facilitate informed user choices. All data are made freely available.

Figures (27)

• Figure 1: We distinguish two categories of censorship: hard censorship (explicit refusal to talk about a topic) and soft censorship (silent omission of a particular viewpoint). Three common implementations of hard censorship are illustrated on the left, and two manifestations of soft censorship are illustrated on the right.
• Figure 2: Taxonomy of different kinds of refusals, suggesting hard censorship.
• Figure 3: Heatmaps showing the refusal rates for each LLM over all political figures.
• Figure 4: Heatmaps showing the different refusal rates for each LLM over all political figures. The panels (from left to right) correspond to error refusals, canned refusals, and generated refusals respectively (see Sec. \ref{['subsec:hard_refusal']})
• Figure 5: Heatmap of omitted criminal indicators in political figure descriptions. This figure shows the normalized frequency with which LLMs omit mentions of criminal activities when queried in English.