Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations

Manon Berriche, Célia Nouri, Chloé Clavel, Jean-Philippe Cointet

TL;DR

SPOT introduces a large, annotated French Facebook corpus to study stopping points—ordinary, context-dependent interventions that pause or redirect online conversations. It operationalizes stopping points as a binary task and benchmarks encoder-based CamemBERT models against instruction-tuned LLMs across multiple prompting strategies, showing supervised encoders outperform LLMs by over 10 F1 points and that contextual metadata further improves performance. The dataset includes 43,305 comments tied to posts, articles, and pages, with robust inter-annotator reliability and a detailed annotation guidelines repository for reproducibility. Findings highlight the importance of conversational and publication context for detecting nuanced social interactions, and they point toward graph-based or hierarchical models and cross-platform extensions as promising directions for future work.

Abstract

We introduce SPOT (Stopping Points in Online Threads), the first annotated corpus translating the sociological concept of stopping point into a reproducible NLP task. Stopping points are ordinary critical interventions that pause or redirect online discussions through a range of forms (irony, subtle doubt or fragmentary arguments) that frameworks like counterspeech or social correction often overlook. We operationalize this concept as a binary classification task and provide reliable annotation guidelines. The corpus contains 43,305 manually annotated French Facebook comments linked to URLs flagged as false information by social media users, enriched with contextual metadata (article, post, parent comment, page or group, and source). We benchmark fine-tuned encoder models (CamemBERT) and instruction-tuned LLMs under various prompting strategies. Results show that fine-tuned encoders outperform prompted LLMs in F1 score by more than 10 percentage points, confirming the importance of supervised learning for emerging non-English social media tasks. Incorporating contextual metadata further improves encoder models F1 scores from 0.75 to 0.78. We release the anonymized dataset, along with the annotation guidelines and code in our code repository, to foster transparency and reproducible research.

SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations

TL;DR

SPOT introduces a large, annotated French Facebook corpus to study stopping points—ordinary, context-dependent interventions that pause or redirect online conversations. It operationalizes stopping points as a binary task and benchmarks encoder-based CamemBERT models against instruction-tuned LLMs across multiple prompting strategies, showing supervised encoders outperform LLMs by over 10 F1 points and that contextual metadata further improves performance. The dataset includes 43,305 comments tied to posts, articles, and pages, with robust inter-annotator reliability and a detailed annotation guidelines repository for reproducibility. Findings highlight the importance of conversational and publication context for detecting nuanced social interactions, and they point toward graph-based or hierarchical models and cross-platform extensions as promising directions for future work.

Abstract

We introduce SPOT (Stopping Points in Online Threads), the first annotated corpus translating the sociological concept of stopping point into a reproducible NLP task. Stopping points are ordinary critical interventions that pause or redirect online discussions through a range of forms (irony, subtle doubt or fragmentary arguments) that frameworks like counterspeech or social correction often overlook. We operationalize this concept as a binary classification task and provide reliable annotation guidelines. The corpus contains 43,305 manually annotated French Facebook comments linked to URLs flagged as false information by social media users, enriched with contextual metadata (article, post, parent comment, page or group, and source). We benchmark fine-tuned encoder models (CamemBERT) and instruction-tuned LLMs under various prompting strategies. Results show that fine-tuned encoders outperform prompted LLMs in F1 score by more than 10 percentage points, confirming the importance of supervised learning for emerging non-English social media tasks. Incorporating contextual metadata further improves encoder models F1 scores from 0.75 to 0.78. We release the anonymized dataset, along with the annotation guidelines and code in our code repository, to foster transparency and reproducible research.

Paper Structure

This paper contains 36 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work and Conceptual Framework
  3. Studies on User Critical Interventions
  4. Related concepts.
  5. Related datasets.
  6. Limits.
  7. Conceptual Foundations and Definition of Stopping Point
  8. SPOT Corpus
  9. Data Collection
  10. Exploratory Observations and Annotation Guidelines
  11. Guidelines Validation, Annotation and Inter-Rater Reliability
  12. Corpus Description and Availability
  13. Classification Task
  14. Models
  15. Encoder-based Models.
  16. ...and 21 more sections

Figures (1)

  • Figure 1: An example thread from the SPOT corpus showing (a) the post with top-level comments, and (b) the reply thread under the first comment. Colors indicate page/group name (orange), domain name (pink), article title (green), non-stopping points (blue), and stopping points (red). English translations for (a) and (b) are provided below.