OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation

TL;DR

This work defines Automated Online Public Opinion Report Generation OPOR-Gen as a formal task and introduces OPOR-Bench, a large event-centric multi-source benchmark, and OPOR-Eval, an agent-based evaluation framework that aligns well with human judgments. Through extensive experiments on frontier LLMs with different generation strategies, the study reveals persistent challenges in temporal reasoning and multi-document synthesis, along with evaluator biases that affect scoring. The findings support a hybrid generation approach and point to future directions that enhance structured, long-form reporting for crisis management. Overall, OPOR-Bench and OPOR-Eval provide a solid foundation for advancing automated crisis reporting and evaluation research.

Abstract

Online Public Opinion Reports consolidate news and social media for timely crisis management by governments and enterprises. While large language models have made automated report generation technically feasible, systematic research in this specific area remains notably absent, particularly lacking formal task definitions and corresponding benchmarks. To bridge this gap, we define the Automated Online Public Opinion Report Generation (OPOR-GEN) task and construct OPOR-BENCH, an event-centric dataset covering 463 crisis events with their corresponding news articles, social media posts, and a reference summary. To evaluate report quality, we propose OPOR-EVAL, a novel agent-based framework that simulates human expert evaluation by analyzing generated reports in context. Experiments with frontier models demonstrate that our framework achieves high correlation with human judgments. Our comprehensive task definition, benchmark dataset, and evaluation framework provide a solid foundation for future research in this critical domain.

Figures (13)

• Figure 1: (a) Traditional methods require manual information consolidation from diverse sources (e.g., news, social media) and labor-intensive report writing and evaluation. In contrast, our (b) automated approach generates and evaluates reports automatically, significantly accelerating the feedback loop.
• Figure 2: Overview of OPOR-Bench construction pipeline. (a) Event-Centric Corpus Construction: Starting from authoritative databases (EM-DAT) and curated lists (Wikipedia), we identify crisis events and collect corresponding multi-source documents—news articles from Wikipedia references and social media posts from X/Twitter API. (b) Dataset Annotation: Three-layered annotation process transforms raw documents into structured data. Human experts annotate timeline phases, while our LLM framework handles factual attribute extraction and social media author classification, ultimately producing a comprehensive reference for each event.
• Figure 3: The distribution of crisis event types in our dataset. The inner ring shows the top-level classification into Natural Disasters (44.9%) and Human-caused Crises (55.1%). The outer ring displays the breakdown into more specific sub-categories.
• Figure 4: The OPOR-Eval architecture: An evaluation agent manages three specialized tools (Fact-Checker, Opinion-Miner, Solution-Counselor) through structured task assignment.
• Figure 5: Comparison of two OPOR-Gen strategies. Left: Modular generation decomposes the task into five sequential subtasks (title, summary, timeline, focus, and suggestions), with each component generated independently. Right: End-to-end generation produces all five report components simultaneously in a single pass, maintaining global coherence throughout the document.