Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

Yixin Nie, Lin Guan, Zhongyao Ma, Anchit Gupta, Yipin Zhou, Xiao Li, Zhengping Zhou, Raymond Zeng, Gelin Zhou, Shigan Chu, Ajay Thampi, Wancen Mu, Nathan Shuster, Ketong Wang, Lin Chen, Jason Brewer, Derek Hao Hu, Alexander McCauley, Jason Weston, Sem Park, Na Zhang, Kevin Tang

TL;DR

The CharacterFlywheel process which integrates data curation, reward modeling, and supervised fine-tuning to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step is detailed.

Abstract

This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1, we refined models across 15 generations using data from both internal and external real-user traffic. Through continuous deployments from July 2024 to April 2025, we conducted controlled 7-day A/B tests showing consistent engagement improvements: 7 of 8 newly deployed models demonstrated positive lift over the baseline, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth. We also observed substantial gains in steerability, with instruction following increasing from 59.2% to 84.8% and instruction violations decreasing from 26.6% to 5.8%. We detail the CharacterFlywheel process which integrates data curation, reward modeling to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step. We also discuss our methods for overfitting prevention and navigating production dynamics at scale. These contributions advance the scientific rigor and understanding of LLMs in social applications serving millions of users.

CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

TL;DR

The CharacterFlywheel process which integrates data curation, reward modeling, and supervised fine-tuning to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step is detailed.

Abstract

This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1, we refined models across 15 generations using data from both internal and external real-user traffic. Through continuous deployments from July 2024 to April 2025, we conducted controlled 7-day A/B tests showing consistent engagement improvements: 7 of 8 newly deployed models demonstrated positive lift over the baseline, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth. We also observed substantial gains in steerability, with instruction following increasing from 59.2% to 84.8% and instruction violations decreasing from 26.6% to 5.8%. We detail the CharacterFlywheel process which integrates data curation, reward modeling to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step. We also discuss our methods for overfitting prevention and navigating production dynamics at scale. These contributions advance the scientific rigor and understanding of LLMs in social applications serving millions of users.
Paper Structure (44 sections, 13 equations, 10 figures, 10 tables, 1 algorithm)

This paper contains 44 sections, 13 equations, 10 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Development Cycle
  4. Data Curation and Annotation
  5. Curation Pipeline
  6. Annotation
  7. Reward Models
  8. Preference Models
  9. User Signal Models
  10. Fine-Tuning and Alignment
  11. Rejection Sampling
  12. SFT and DPO
  13. Reinforcement Learning
  14. Stylistic Artifact Mitigation
  15. Evaluation and Feedback Loop
  16. ...and 29 more sections

Figures (10)

  • Figure 1: Character Creation Interface.
  • Figure 2: Figure \ref{['fig:sub1']} illustrates our cumulative model improvement, analogous to iteratively climbing the engagingness landscape. Figures \ref{['fig:sub2']}, \ref{['fig:sub3']}, and \ref{['fig:sub4']} show data sampling, contour interpolating, and model updating, respectively.
  • Figure 3: The CharacterFlywheel Iterative Development Cycle.
  • Figure 4: The Data Pipeline.
  • Figure 5: Pre-launch quality progression (V2-V7). Left panel: Human win rates comparing CharacterFlywheel models against GPT-4o baseline, showing improvement from 37.4% (V3) to 46.2% (V7). Right panel: Win rates against the immediate previous model version, displaying both human evaluations (blue bars, 50.2%-52.5%) and reward model predictions (orange bars, 53.6%-57.6%). Both signals consistently exceed the 50% neutral threshold (dashed line), indicating quality improvements across iterations.
  • ...and 5 more figures