Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Jiayu Wang, Junyoung Lee

Abstract

As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's $α$) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Abstract

As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's ) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.

Paper Structure

This paper contains 18 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Rubric Development
  4. Dataset Curation
  5. Data Augmentation and Regularization
  6. Training Pipeline
  7. Baselines
  8. Results
  9. Generalizability to Other Annotation Tasks
  10. Discussion
  11. Conclusion
  12. Appendix
  13. LLM Response Evaluation Rubric Description
  14. SLMs used for SPS dataset curation
  15. Example of Prompt for Finetuning on SPS Dataset
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Hierarchical structure of the proposed evaluation rubric.
  • Figure 2: Plot of Training and Validation loss without using Data Augmentation
  • Figure 3: Plot of Training and Validation loss using Proposed Data Augmentation
  • Figure 4: Plot of Training and Validation loss using LoRA Dropout