Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fane at SemEval-2025 Task 10: Zero-Shot Entity Framing with Large Language Models

Enfa Fane, Mihai Surdeanu, Eduardo Blanco, Steven R. Corman

TL;DR

This work systematically evaluates zero-shot large language models for entity framing in news using a hierarchical two-stage prompting approach. By exploring input contexts, prompt designs, and inference strategies, the authors show that decomposing the task into identifying broad main roles first and refining into fine-grained roles yields stronger performance than single-step classification. Key contributions include a modular prompting framework, evidence that framing-preserved summaries and entity-focused contexts serve different classification levels, and the finding that carefully engineered prompts can let smaller models rival larger ones. The results demonstrate practical implications for prompt design and context optimization in zero-shot framing tasks, achieving competitive performance on SemEval-2025 Task 10 and providing a publicly available codebase for reproducibility.

Abstract

Understanding how news narratives frame entities is crucial for studying media's impact on societal perceptions of events. In this paper, we evaluate the zero-shot capabilities of large language models (LLMs) in classifying framing roles. Through systematic experimentation, we assess the effects of input context, prompting strategies, and task decomposition. Our findings show that a hierarchical approach of first identifying broad roles and then fine-grained roles, outperforms single-step classification. We also demonstrate that optimal input contexts and prompts vary across task levels, highlighting the need for subtask-specific strategies. We achieve a Main Role Accuracy of 89.4% and an Exact Match Ratio of 34.5%, demonstrating the effectiveness of our approach. Our findings emphasize the importance of tailored prompt design and input context optimization for improving LLM performance in entity framing.

Fane at SemEval-2025 Task 10: Zero-Shot Entity Framing with Large Language Models

TL;DR

This work systematically evaluates zero-shot large language models for entity framing in news using a hierarchical two-stage prompting approach. By exploring input contexts, prompt designs, and inference strategies, the authors show that decomposing the task into identifying broad main roles first and refining into fine-grained roles yields stronger performance than single-step classification. Key contributions include a modular prompting framework, evidence that framing-preserved summaries and entity-focused contexts serve different classification levels, and the finding that carefully engineered prompts can let smaller models rival larger ones. The results demonstrate practical implications for prompt design and context optimization in zero-shot framing tasks, achieving competitive performance on SemEval-2025 Task 10 and providing a publicly available codebase for reproducibility.

Abstract

Understanding how news narratives frame entities is crucial for studying media's impact on societal perceptions of events. In this paper, we evaluate the zero-shot capabilities of large language models (LLMs) in classifying framing roles. Through systematic experimentation, we assess the effects of input context, prompting strategies, and task decomposition. Our findings show that a hierarchical approach of first identifying broad roles and then fine-grained roles, outperforms single-step classification. We also demonstrate that optimal input contexts and prompts vary across task levels, highlighting the need for subtask-specific strategies. We achieve a Main Role Accuracy of 89.4% and an Exact Match Ratio of 34.5%, demonstrating the effectiveness of our approach. Our findings emphasize the importance of tailored prompt design and input context optimization for improving LLM performance in entity framing.
Paper Structure (36 sections, 3 figures, 11 tables)

This paper contains 36 sections, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background: Task & Dataset
  3. Related Work
  4. Prompting for Framing Classification
  5. Input Context Variation
  6. Prompt Design
  7. Inference Strategy
  8. Experimental Setup
  9. Single-Label Approximation for Fine-Grained Role Classification
  10. Evaluation Metrics
  11. Model and API
  12. Results
  13. Development
  14. Single-Step Approach
  15. Multi-Step Approach
  16. ...and 21 more sections

Figures (3)

  • Figure 1: An example document excerpt from the dataset. The author strategically uses loaded language, contrastive framing, and selective emphasis to shape the reader’s perception of Bill Gates as corrupt and deceptive. By highlighting contradictions in his public advocacy, casting doubt on the authenticity of his media appearances, and referencing controversial associations, the text reinforces a negative framing.
  • Figure 2: Structure of the prompt template. Curly-bracketed content is dynamically replaced based on the experimental setting (see Appendix \ref{['sec:prompt-template-details']} for exact phrasing). The template is designed to minimize variability, ensuring that observed differences arise solely from targeted prompt modifications.
  • Figure 3: Prompt used to generate neutral summaries. The instructions guide the LLM to ensure that that the entity’s role is explicitly introduced while maintaining a factual and impartial tone.