OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants

Abstract

Warning: Contents of this paper may be upsetting. Public attitudes towards key societal issues, expressed on online media, are of immense value in policy and reform efforts, yet challenging to understand at scale. We study one such social issue: homelessness in the U.S., by leveraging the remarkable capabilities of large language models to assist social work experts in analyzing millions of posts from Twitter. We introduce a framing typology: Online Attitudes Towards Homelessness (OATH) Frames: nine hierarchical frames capturing critiques, responses and perceptions. We release annotations with varying degrees of assistance from language models, with immense benefits in scaling: 6.5x speedup in annotation time while only incurring a 3 point F1 reduction in performance with respect to the domain experts. Our experiments demonstrate the value of modeling OATH-Frames over existing sentiment and toxicity classifiers. Our large-scale analysis with predicted OATH-Frames on 2.4M posts on homelessness reveal key trends in attitudes across states, time periods and vulnerable populations, enabling new insights on the issue. Our work provides a general framework to understand nuanced public attitudes at scale, on issues beyond homelessness.

Figures (15)

• Figure 1: Overview of OATH-Frames Collection. (1.) Domain experts applied grounded theory to surface nine Issue-specific frames, their corresponding definitions (OATH-Definitions), and annotation guidelines (OATH-Guidelines). (2.) We annotated posts with OATH-Frames via Experts, via Experts + LLM (GPT-4), and via Multlilabel Classifier (Flan-T5-Large). Our Experts + LLM annotations pipeline consists of (2.1) prompt editing based on insights from domain experts and chains-of-thought, and (2.2) validation of predicted frames. (3.) We analyzed annotated posts for attitudes across social and political dimensions.
• Figure 2: Proportion of Issue-specific frames across different annotation methods. Distribution of OATH-Frames across annotation strategies is similar. See more details in \ref{['tab:appendix:annotation_splits']} in \ref{['sec:appendix:prediction']}.
• Figure 3: Experts+LLM-Prompt Editing: We prompt GPT-4 with our task, OATH-Definitions, and our instruction as well as our own OATH-Guidelines and GPT-4's Chain-of-Thought reasoning wei2022chain, in an iterative setup. We observe that CoT serves as a signal to understanding misinterpretations of OATH-Definitions and benefits from addition of expert annotator guidelines (OATH-Guidelines).
• Figure 4: Distributions of sentiment and toxicity scores for posts labeled with and without HarmGen. in our subset of 4.1K Expert-annotated posts.
• Figure 5: Linear regression results for state level factors: cost of living and percent of unsheltered PEH, predicting proportion of frames for each state. The shaded region represents the standard error of the fit to the true values.