Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision

Pritam Kadasi, Abhishek Upperwal, Mayank Singh

TL;DR

This work tackles the problem of weak instruction definitions in instruction tuning by introducing Task--Specificity Score (TSS), a model-based, instance-level measure of how much an instruction actually constrains the desired output among plausible alternatives. It further extends to TSS++, a quality-aware, contrastive variant that leverages hard negatives to robustly assess specificity. Through extensive experiments on Alpaca, Dolly-15k, and NI-20 corpora with open models like Gemma, LLaMA, and Qwen, the authors show that prioritizing high-specificity examples improves data efficiency under budget constraints and often complements traditional quality-based filters such as perplexity and IFD. The results demonstrate that instruction specificity is a distinct, practically useful signal for data curation in instruction tuning, enabling better generalization under limited budgets and guiding future data-selection strategies.

Abstract

Instruction tuning is now the default way to train and adapt large language models, but many instruction--input--output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: \emph{does the instruction uniquely determine the target output?} We propose the \textbf{Task--Specificity Score (TSS)} to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce \textbf{TSS++}, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction datasets (\textsc{Alpaca}, \textsc{Dolly-15k}, \textsc{NI-20}) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight token budgets and complements quality-based filters such as perplexity and IFD.

Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision

TL;DR

This work tackles the problem of weak instruction definitions in instruction tuning by introducing Task--Specificity Score (TSS), a model-based, instance-level measure of how much an instruction actually constrains the desired output among plausible alternatives. It further extends to TSS++, a quality-aware, contrastive variant that leverages hard negatives to robustly assess specificity. Through extensive experiments on Alpaca, Dolly-15k, and NI-20 corpora with open models like Gemma, LLaMA, and Qwen, the authors show that prioritizing high-specificity examples improves data efficiency under budget constraints and often complements traditional quality-based filters such as perplexity and IFD. The results demonstrate that instruction specificity is a distinct, practically useful signal for data curation in instruction tuning, enabling better generalization under limited budgets and guiding future data-selection strategies.

Abstract

Instruction tuning is now the default way to train and adapt large language models, but many instruction--input--output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: \emph{does the instruction uniquely determine the target output?} We propose the \textbf{Task--Specificity Score (TSS)} to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce \textbf{TSS++}, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction datasets (\textsc{Alpaca}, \textsc{Dolly-15k}, \textsc{NI-20}) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight token budgets and complements quality-based filters such as perplexity and IFD.
Paper Structure (42 sections, 3 equations, 2 figures, 5 tables, 2 algorithms)

This paper contains 42 sections, 3 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction tuning and mixture construction.
  4. Data filtering/selection for instruction tuning.
  5. Measuring instruction utility beyond quality.
  6. Method: Task--Specificity Score ($\mathrm{TSS}$)
  7. Setup.
  8. Motivation.
  9. Generating plausible alternative instructions
  10. Rationale.
  11. Task--Specificity Score ($\mathrm{TSS}$)
  12. Interpretation.
  13. Information-Theoretic view
  14. Using $\mathrm{TSS}$ for Budgeted Filtering and Weighting
  15. Filtering under a token budget.
  16. ...and 27 more sections

Figures (2)

  • Figure 1: Budget sweeps. X-axis: retention fraction $\rho \in \{0.05,0.15,0.45,0.75,1.0\}$. Y-axis: SUM. Plot one curve per method (Random, PPL, IFD, $\mathrm{TSS}$, $\mathrm{TSS}$ (E), $\mathrm{TSS{+}{+}}$, $\mathrm{TSS{+}{+}}$ (E)).
  • Figure 2: Winner map across budgets. A heatmap with rows as (model, dataset) and columns as $\rho$; each cell indicates the winning method family (Random vs Quality vs $\mathrm{TSS}$ vs $\mathrm{TSS{+}{+}}$).