Large Language Models as Mirrors of Societal Moral Standards
Evi Papadopoulou, Hadi Mohammadi, Ayoub Bagheri
TL;DR
This paper investigates whether pre-trained language models encode cross-cultural moral norms, focusing on issues such as homosexuality and divorce. It reformulates questions from the World Values Survey (WVS) and the PEW Global Attitudes Survey as prompts and computes a moral score by $language_model_score = moral_logprob - nonmoral_logprob$, averaged across token pairs. Across monolingual and multilingual models from HuggingFace, the study finds limited alignment with human norms, with BLOOMZ-560M showing the strongest cross-cultural signal but still falling short of accurately capturing nuanced moral beliefs. The results underscore the need for culturally aware AI systems and point to data, prompting, and methodology improvements to better reflect universal human values.
Abstract
Prior research has demonstrated that language models can, to a limited extent, represent moral norms in a variety of cultural contexts. This research aims to replicate these findings and further explore their validity, concentrating on issues like 'homosexuality' and 'divorce'. This study evaluates the effectiveness of these models using information from two surveys, the WVS and the PEW, that encompass moral perspectives from over 40 countries. The results show that biases exist in both monolingual and multilingual models, and they typically fall short of accurately capturing the moral intricacies of diverse cultures. However, the BLOOM model shows the best performance, exhibiting some positive correlations, but still does not achieve a comprehensive moral understanding. This research underscores the limitations of current PLMs in processing cross-cultural differences in values and highlights the importance of developing culturally aware AI systems that better align with universal human values.