Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

TL;DR

This paper presents a low-cost, LLM-driven method to evaluate the potential efficacy of regulatory policies in mitigating AI-induced harms. By generating scenario pairs that depict impacts before and after a policy (Article 50 of the EU AI Act) and translating these narratives into quantitative judgments across four risk dimensions, the authors create a human-centric, anticipatory governance workflow anchored to a formal impact taxonomy. A case study in the media information ecosystem shows that the policy is perceived to reduce severity and reach in several domains (notably autonomy, labor, media quality, and well-being) while being less effective in others (education, security, social cohesion). The work demonstrates the feasibility of using LLMs for scenario-based policy brainstorming and early-stage evaluation, offering a potential tool for policymakers and researchers to iteratively explore mitigation strategies before costly implementations. The findings underscore both the promise and the limitations of current LLM capabilities in faithfully simulating complex policy-jeopardized futures and highlight areas for methodological refinement and broader application.

Abstract

The rapid advancement of AI technologies yields numerous future impacts on individuals and society. Policymakers are tasked to react quickly and establish policies that mitigate those impacts. However, anticipating the effectiveness of policies is a difficult task, as some impacts might only be observable in the future and respective policies might not be applicable to the future development of AI. In this work we develop a method for using large language models (LLMs) to evaluate the efficacy of a given piece of policy at mitigating specified negative impacts. We do so by using GPT-4 to generate scenarios both pre- and post-introduction of policy and translating these vivid stories into metrics based on human perceptions of impacts. We leverage an already established taxonomy of impacts of generative AI in the media environment to generate a set of scenario pairs both mitigated and non-mitigated by the transparency policy in Article 50 of the EU AI Act. We then run a user study (n=234) to evaluate these scenarios across four risk-assessment dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. We find that this transparency legislation is perceived to be effective at mitigating harms in areas such as labor and well-being, but largely ineffective in areas such as social cohesion and security. Through this case study we demonstrate the efficacy of our method as a tool to iterate on the effectiveness of policy for mitigating various negative impacts. We expect this method to be useful to researchers or other stakeholders who want to brainstorm the potential utility of different pieces of policy or other mitigation strategies.

Figures (5)

• Figure 1: Flow diagram of study, starting with prompt engineering to generate 150 scenario pairs (3 scenarios for each of 50 impact types), then manual author validation check removed 33 scenario pairs (11 impact types), then had human raters evaluate 117 scenarios, and finally analyzed the aggregated data across impact themes.
• Figure 2: Bar chart displaying from top to bottom the four dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. Left plot: we first display the mean raw scores ($M_{S}$ and $M_{S'}$) for each impact theme (in alphabetical order: overall, autonomy, education, labor, legal rights, media quality, political, security, social cohesion, trustworthiness, and well-being). Right plot: we display the deltas ($M_{S}-M_{S'}$) for each theme. For significance levels we include above the bars: * for $p\leq0.05$, ** for $p\leq0.01$, and *** for $p\leq0.001$.
• Figure 3: Example scenario ($S$) for accuracy and errors within media quality.
• Figure 4: Example re-written scenario ($S'$) for accuracy and errors within media quality after transparency legislation was introduced.
• Figure 5: Heat map displaying how many times human-identified specific impact types from kieslich_anticipating_2023 were present in 10 LLM generated scenarios for each general impact theme. We prompted GPT-4 to generate a scenario 10 times for each of the 10 impact themes: autonomy, education, labor, legal rights, media quality, political, security, social cohesion, trustworthiness, and well-being. We then had two of the authors independently analyze whether each of the specific impact types were present in the generated scenarios. Darker colors correspond to impact types that were present more times than those with lighter colors. White backgrounds correspond to impact types that were never present in the scenarios generated by prompts focusing on the general impact theme.