Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale

Karl Gustav Gailit, Kadri Muischnek, Kairit Sirts

TL;DR

The paper presents the creation of a 1000-text Estonian document-level subjectivity dataset annotated on a continuous 0–100 scale by four human raters, complemented by GPT-5 annotations. It reveals moderate inter-annotator agreement, explored through a secondary re-annotation of highly disagreed items and an additional fourth annotator to stabilize scores. A thorough evaluation of GPT-5 demonstrates plausible yet non-identical alignment with human judgments, highlighting systematic differences in handling quotes and colloquial language. The work establishes a valuable resource for Estonian NLP and illustrates both the feasibility and current limits of automated subjectivity scoring for downstream applications.

Abstract

This article presents the creation of an Estonian-language dataset for document-level subjectivity, analyzes the resulting annotations, and reports an initial experiment of automatic subjectivity analysis using a large language model (LLM). The dataset comprises of 1,000 documents-300 journalistic articles and 700 randomly selected web texts-each rated for subjectivity on a continuous scale from 0 (fully objective) to 100 (fully subjective) by four annotators. As the inter-annotator correlations were moderate, with some texts receiving scores at the opposite ends of the scale, a subset of texts with the most divergent scores was re-annotated, with the inter-annotator correlation improving. In addition to human annotations, the dataset includes scores generated by GPT-5 as an experiment on annotation automation. These scores were similar to human annotators, however several differences emerged, suggesting that while LLM based automatic subjectivity scoring is feasible, it is not an interchangeable alternative to human annotation, and its suitability depends on the intended application.

Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale

TL;DR

The paper presents the creation of a 1000-text Estonian document-level subjectivity dataset annotated on a continuous 0–100 scale by four human raters, complemented by GPT-5 annotations. It reveals moderate inter-annotator agreement, explored through a secondary re-annotation of highly disagreed items and an additional fourth annotator to stabilize scores. A thorough evaluation of GPT-5 demonstrates plausible yet non-identical alignment with human judgments, highlighting systematic differences in handling quotes and colloquial language. The work establishes a valuable resource for Estonian NLP and illustrates both the feasibility and current limits of automated subjectivity scoring for downstream applications.

Abstract

This article presents the creation of an Estonian-language dataset for document-level subjectivity, analyzes the resulting annotations, and reports an initial experiment of automatic subjectivity analysis using a large language model (LLM). The dataset comprises of 1,000 documents-300 journalistic articles and 700 randomly selected web texts-each rated for subjectivity on a continuous scale from 0 (fully objective) to 100 (fully subjective) by four annotators. As the inter-annotator correlations were moderate, with some texts receiving scores at the opposite ends of the scale, a subset of texts with the most divergent scores was re-annotated, with the inter-annotator correlation improving. In addition to human annotations, the dataset includes scores generated by GPT-5 as an experiment on annotation automation. These scores were similar to human annotators, however several differences emerged, suggesting that while LLM based automatic subjectivity scoring is feasible, it is not an interchangeable alternative to human annotation, and its suitability depends on the intended application.

Paper Structure

This paper contains 29 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Creation of the Estonian Subjectivity Dataset
  3. Dataset Creation Principles
  4. Pilot Project
  5. Documents Included
  6. Annotation Process
  7. Annotation Process
  8. Annotation Results
  9. Secondary Annotation
  10. Re-annotation
  11. Results
  12. Qualitative Insights
  13. Fourth Annotator
  14. LLM Annotation
  15. Initial trial
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Histogram of the subjectivity scores given by the three annotators
  • Figure 2: Inter-annotator agreement between categories on the full dataset of 1000 texts.
  • Figure 3: Histogram of differences between each of the annotator pairs. Higher x-axis values indicated higher differences in annotators' scores.
  • Figure 4: The change in annotators' scores between the first and second annotation. No difference: initial difference between annotators < 3 points; Moderate difference: initial difference 24--26; High difference: initial difference > 50.
  • Figure 5: Instructions used to prompt GPT-5.