Benchmarking LLMs for Political Science: A United Nations Perspective

TL;DR

This work introduces UNBench, the first comprehensive benchmark for evaluating large language models on political science tasks using United Nations Security Council records from 1994 to 2024. It defines four interconnected tasks mapped to the UN drafting, voting, and discussing stages: co-penholder judgement, representative voting simulation, draft adoption prediction, and representative statement generation. Through extensive experiments with diverse models, including GPT-4o and DeepSeek-V3, the study reveals clear strengths and limitations across tasks, highlighting the potential for LLMs to assist in understanding and forecasting multilateral diplomacy while underscoring the need for targeted benchmarks and careful interpretation. UNBench thus advances AI-driven analysis of global governance and provides a foundation for further research at the intersection of AI and international relations.

Abstract

Large Language Models (LLMs) have achieved significant advances in natural language processing, yet their potential for high-stake political decision-making remains largely unexplored. This paper addresses the gap by focusing on the application of LLMs to the United Nations (UN) decision-making process, where the stakes are particularly high and political decisions can have far-reaching consequences. We introduce a novel dataset comprising publicly available UN Security Council (UNSC) records from 1994 to 2024, including draft resolutions, voting records, and diplomatic speeches. Using this dataset, we propose the United Nations Benchmark (UNBench), the first comprehensive benchmark designed to evaluate LLMs across four interconnected political science tasks: co-penholder judgment, representative voting simulation, draft adoption prediction, and representative statement generation. These tasks span the three stages of the UN decision-making process--drafting, voting, and discussing--and aim to assess LLMs' ability to understand and simulate political dynamics. Our experimental analysis demonstrates the potential and challenges of applying LLMs in this domain, providing insights into their strengths and limitations in political science. This work contributes to the growing intersection of AI and political science, opening new avenues for research and practical applications in global governance. The UNBench Repository can be accessed at: https://github.com/yueqingliang1/UNBench.

Figures (9)

• Figure 1: Three Key Stages of the United Nations Decision-Making Process
• Figure 2: The proposed UNBench. It consists of $4$ tasks extracted from different stages of a UN draft.
• Figure 3: Models performance in Task 1 by varying the number of choices.
• Figure 4: Author-Subjects Relationships. This figure shows the co-occurrence matrix of the top 15 authors and subjects. Each cell represents the number of times an author has written about a topic. The darker the cell, the more the author has written about the topic.
• Figure 5: This figure shows the top 30 subjects that at least one country did not vote 'Yes' on.