How can LLMs Support Policy Researchers? Evaluating an LLM-Assisted Workflow for Large-Scale Unstructured Data

Abstract

Policy researchers need scalable ways to surface public views, yet they often rely on interviews, listening sessions, and surveys-analyzed thematically-that are slow, expensive, and limited in scale and diversity. LLMs offer new possibilities for thematic analysis of unstructured text, yet we know little about how LLM-assisted workflows perform for policy research. Building on a workflow for LLM-assisted thematic analysis of online forums, we conduct a study with 11 policy researchers, who use an early prototype and see it as a quick, rough-and-ready input to their research. We then extend and scale the workflow to analyze millions of Reddit posts and 1,058 chatbot-led interview transcripts on a policy-relevant topic, treating these sources as rich and scalable data for policy discourse. We compare the synthesized themes to those from authoritative policy reports, identify points of alignment and divergence, and discuss what this implies for policy researchers adopting LLM-assisted workflows.

Figures (11)

• Figure 1: Overview of the Workflow. The workflow comprises four stages, where the two stages(Quote Extraction and Thematic Analysis) are adapted from an existing LLM-based thematic analysis workflow called QuaLLM. We demonstrated details of each stage in Sections 3.1, 3.2, 3.3, and 3.4 respectively.
• Figure 2: Example of a discussion in r/cscareers subreddit
• Figure 3: Illustration of How User Interact with the Workflow in This Study. Action 1: The user enters a policy-relevant topic (left), prompting the backend (center) to identify relevant datasets, which the system (right) then presents as recommendations. Action 2: The user selects or uploads the dataset(s), triggering the backend to analyze for high‐level themes, and the system outputs theme suggestions. Action 3: The user chooses or defines a final theme, prompting the backend to extract relevant quotes, generate subtopics, and map each quote accordingly, resulting in a downloadable report.
• Figure 4: Screenshots of the User Interface in this study. View 1 shows the interface. This view prompts users to input a research domain for analysis (Action 1) and select a data source (Action 2). View 2 shows high-level themes in the workflow interface. The user can then select or search for a primary research topic (Action 3). View 3 shows the final Report View of the interface, displaying all subtopics identified from the data source and quote counts. View 4 shows examples of 5-6 word summaries beneath the subtopic and a couple of fully displayed quotes on which the summaries are based.
• Figure 5: Timeline of the User Study. This figure illustrates the four-phase research method comparing AI-assisted and participants' own non-AI expert approach. The process begins with an interview and tutorial, followed by two sequential research parts randomized based on topic (Climate Change or Social Media & Kids) and method (with the workflow or own non-AI expert approach), and concludes with a retrospective survey and interview. Each research phase is allocated 10-15 minutes, with participants documenting both quantitative and qualitative findings using standardized interviewee report templates throughout the process.