Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures

Shuangshuang Ying, Yunwen Li, Xingwei Qu, Xin Li, Sheng Jin, Minghao Liu, Zhoufutu Wen, Xeron Du, Tianyu Zheng, Yichi Zhang, Letian Ni, Yuyang Cheng, Qiguang Chen, Jingzhe Ding, Shengda Long, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Libo Qin, Ge Zhang, Wenhao Huang, Wanxiang Che, Chenghua Lin

TL;DR

The results suggest that current RLHF methods primarily learn to detect objective errors rather than capture subjective quality preferences, and that successful preference modeling may require intermediate reasoning representations rather than direct classification.

Abstract

Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed. We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres, where responses are matched for objective correctness, factual accuracy, and length. On this benchmark, sequence-based reward models--the standard architecture for RLHF--achieve only 52.7% mean accuracy, while zero-shot language model judges perform at 53.9%. In contrast, generative reward models that produce explicit reasoning chains achieve 81.8% accuracy. We observe high within-model variance across genres: individual models range from 18.2% to 81.8% accuracy across different writing categories, with standard deviations averaging 10.1%. This variance persists regardless of model scale, with 27B parameter models showing no consistent improvement over 8B variants. Our results suggest that current RLHF methods primarily learn to detect objective errors rather than capture subjective quality preferences (e.g., creativity, stylistic flair, and emotional resonance), and that successful preference modeling may require intermediate reasoning representations rather than direct classification.

Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures

TL;DR

The results suggest that current RLHF methods primarily learn to detect objective errors rather than capture subjective quality preferences, and that successful preference modeling may require intermediate reasoning representations rather than direct classification.

Abstract

Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed. We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres, where responses are matched for objective correctness, factual accuracy, and length. On this benchmark, sequence-based reward models--the standard architecture for RLHF--achieve only 52.7% mean accuracy, while zero-shot language model judges perform at 53.9%. In contrast, generative reward models that produce explicit reasoning chains achieve 81.8% accuracy. We observe high within-model variance across genres: individual models range from 18.2% to 81.8% accuracy across different writing categories, with standard deviations averaging 10.1%. This variance persists regardless of model scale, with 27B parameter models showing no consistent improvement over 8B variants. Our results suggest that current RLHF methods primarily learn to detect objective errors rather than capture subjective quality preferences (e.g., creativity, stylistic flair, and emotional resonance), and that successful preference modeling may require intermediate reasoning representations rather than direct classification.
Paper Structure (49 sections, 1 equation, 4 figures, 3 tables)

This paper contains 49 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. WritingPreferenceBench
  3. Benchmark Construction Pipeline
  4. Dataset Statistics
  5. Experiments
  6. Evaluation Protocols
  7. Models
  8. Implementation Details
  9. Results
  10. Reward Model Performance
  11. Sequence classifiers fail on subjective preference.
  12. Generative architecture enables strong performance.
  13. Scale improves stability, not just accuracy.
  14. Sequence classifiers exhibit catastrophic genre failure.
  15. Language Model Judge Performance
  16. ...and 34 more sections

Figures (4)

  • Figure 1: WritingPreferenceBench isolates subjective writing quality by neutralizing objective confounds (grammar, factuality, length). Across 1,800 human-validated preference pairs, standard sequence classifiers (SC-RM) perform near-randomly while generative reward models (GenRM) achieve 30% higher accuracy—but both architectures exhibit catastrophic instability across genres, exposing the brittleness of current preference learning.
  • Figure 2: The data curation pipeline of WritingPreferenceBench. Our multi-stage process begins with expert-crafted queries across 51 genres, generates diverse responses using 20 state-of-the-art models, and culminates in rigorous human evaluation by trained annotators. Quality control mechanisms operate throughout to ensure preference pairs reflect genuine subjective quality distinctions rather than objective differences.
  • Figure 3: Distribution of preference pairs across a sample of the 8 writing macro-categories for both English and Chinese in WritingPreferenceBench. The dataset maintains balanced coverage across diverse writing genres, with deliberate oversampling of underrepresented categories to ensure comprehensive evaluation of preference modeling capabilities.
  • Figure 4: Length distributions of chosen and rejected responses reveal the statistical signature of creative quality in WritingPreferenceBench. Distributions truncated at 6K for visualization; dashed lines indicate means.