Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WRAVAL -- WRiting Assist eVALuation

Gabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine

TL;DR

WRAVAL introduces a task-focused evaluation framework for Writing Assistance by small language models, addressing a gap where existing benchmarks emphasize reasoning tasks. The framework combines synthetic data generation, flexible inference with diverse backends, and LLM- and human-based evaluation to assess tone-specific rewriting across edge and private computing settings. Key contributions include a modular four-step pipeline, a taxonomy of nine rewrite tones, and evidence that well-tuned SLMs can approach LLM performance on WA tasks, while highlighting areas where LLMs struggle in non-reasoning contexts. The work provides practical tools and benchmarks for practitioners to evaluate WA capabilities on-device, enabling scalable, privacy-preserving deployment and guiding future improvements in finetuning, prompting, and evaluation protocols.

Abstract

The emergence of Large Language Models (LLMs) has shifted language model evaluation toward reasoning and problem-solving tasks as measures of general intelligence. Small Language Models (SLMs) -- defined here as models under 10B parameters -- typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs' effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs' capabilities in non-reasoning tasks where predefined evaluation datasets don't exist. Our framework combines novel approaches in data generation, prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval.

WRAVAL -- WRiting Assist eVALuation

TL;DR

WRAVAL introduces a task-focused evaluation framework for Writing Assistance by small language models, addressing a gap where existing benchmarks emphasize reasoning tasks. The framework combines synthetic data generation, flexible inference with diverse backends, and LLM- and human-based evaluation to assess tone-specific rewriting across edge and private computing settings. Key contributions include a modular four-step pipeline, a taxonomy of nine rewrite tones, and evidence that well-tuned SLMs can approach LLM performance on WA tasks, while highlighting areas where LLMs struggle in non-reasoning contexts. The work provides practical tools and benchmarks for practitioners to evaluate WA capabilities on-device, enabling scalable, privacy-preserving deployment and guiding future improvements in finetuning, prompting, and evaluation protocols.

Abstract

The emergence of Large Language Models (LLMs) has shifted language model evaluation toward reasoning and problem-solving tasks as measures of general intelligence. Small Language Models (SLMs) -- defined here as models under 10B parameters -- typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs' effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs' capabilities in non-reasoning tasks where predefined evaluation datasets don't exist. Our framework combines novel approaches in data generation, prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval.
Paper Structure (28 sections, 1 figure, 3 tables)

This paper contains 28 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SLMs on non-reasoning tasks
  4. Evaluation Frameworks
  5. Evaluation Datasets
  6. Finetuning
  7. Design
  8. Framework – WRAVAL: Writing Assistant eVALuation
  9. generate
  10. inference
  11. llm judge
  12. human judge
  13. lm router
  14. prompt directory
  15. Typical usage
  16. ...and 13 more sections

Figures (1)

  • Figure 1: WRAVAL – Writing Assistant eVALuation pipeline. Our proposed framework (1) generates data with custom prompts at scale, (2) performs inference with models hosted on Bedrock / Ollama / Sagemaker Endpoints, (3) runs A bigger LLM-as-a-judge on all inputs-output pairs, (4) transforms the data into a format ready for human judges. A symbol hovers over each step of the pipeline to indicate that it is supported by either our own llm router (bottom left), or Sagemaker Groundtruth (bottom right).