Users Mispredict Their Own Preferences for AI Writing Assistance

TL;DR

Proactive NLG systems must predict user interest in drafting help, but explicit self-reports mislead design. Using a $2 \times 2 \times 2 \times 2$ factorial vignette with 50 participants across 16 email scenarios, the study shows compositional effort ($\rho = 0.597$, $p < 0.05$) drives preferences, while urgency has near-zero predictive power ($\rho \approx 0$), revealing a robust perception–behavior gap. Systems designed from stated preferences underperform behaviorally grounded designs (57.7% vs 61.3%), illustrating the practical costs of relying on introspection. The findings advocate prioritizing implicit behavioral signals and cognitive-load indicators for proactive NLG, with interpretable rule-based weights serving as a reliable deployment path before ML personalization scales.

Abstract

Proactive AI writing assistants need to predict when users want drafting help, yet we lack empirical understanding of what drives preferences. Through a factorial vignette study with 50 participants making 750 pairwise comparisons, we find compositional effort dominates decisions ($ρ= 0.597$) while urgency shows no predictive power ($ρ\approx 0$). More critically, users exhibit a striking perception-behavior gap: they rank urgency first in self-reports despite it being the weakest behavioral driver, representing a complete preference inversion. This misalignment has measurable consequences. Systems designed from users' stated preferences achieve only 57.7\% accuracy, underperforming even naive baselines, while systems using behavioral patterns reach significantly higher 61.3\% ($p < 0.05$). These findings demonstrate that relying on user introspection for system design actively misleads optimization, with direct implications for proactive natural language generation (NLG) systems.

Figures (13)

• Figure 1: Predictive strength of email dimensions measured by Spearman correlation with Bradley-Terry preference strengths. Effort to compose dominates as the only statistically significant predictor, while urgency shows near-zero correlation. Sender importance and email type show weak non-significant trends. *$p<0.05$.
• Figure 2: Key pairwise dimension interactions showing mean Bradley-Terry preference strengths. Effort $\times$ Sender produces the highest preference when both factors align (important sender $\times$ high effort: $\bar{\pi}$ = 0.626), demonstrating a "high-stakes, high-burden" heuristic. Sender $\times$ Type shows sender importance primarily matters for routine emails ($\bar{\pi}$ = 0.599) with minimal effect on novel emails.
• Figure 3: Comparison of stated versus revealed preference rankings. Users report urgency as most important (rank 2.06) but behavioral analysis shows it has no predictive power ($\rho \approx 0$). Effort, the actual strongest driver ($\rho = 0.597$), receives only moderate stated importance (rank 2.18). The non-significant correlation masks a systematic inversion at the extremes, suggesting stated preferences will prioritize the wrong factors.
• Figure 4: Feature importance for preference prediction using Random Forest with all pairwise interactions. Effort main effect (17.8%) is the single most important feature, followed closely by Effort $\times$ Urgency interaction (17.0%). Interactions collectively dominate at 61.7% versus 38.3% for main effects, demonstrating synergistic preference structure. However, models trained with interactions (59.5% accuracy) perform worse than models using only main effects (59.7%), indicating interactions contribute to feature importance without improving predictive performance.
• Figure 5: Informed consent page presented at the beginning of the study. Participants reviewed study purpose, procedures, risks, benefits, and data handling before providing consent to participate.