Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SALSA: Single-pass Autoregressive LLM Structured Classification

Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben Zaken

TL;DR

SALSA tackles the challenge of efficiently turning decoder-only instruction-tuned LLMs into effective text classifiers by mapping each label to a unique output token and performing a single forward pass with structured prompts. The method combines class-to-token mappings, structured prompting, and LoRA-based fine-tuning to achieve strong zero-shot and few-shot performance and fast convergence, while offering an extension to continuous values via discretization. Across GLUE subsets and domain-specific benchmarks (e.g., MedNLI, MedMCQA, HateXplain), SALSA achieves state-of-the-art results and demonstrates robustness and scalability for high-throughput, real-world classification tasks. The approach reduces inference latency and preserves the base model through parameter-efficient adapters, making it well-suited for resource-constrained settings and broad applicability in NLP classification tasks.

Abstract

Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output token, and prompts are constructed to elicit a single-token response. During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.

SALSA: Single-pass Autoregressive LLM Structured Classification

TL;DR

SALSA tackles the challenge of efficiently turning decoder-only instruction-tuned LLMs into effective text classifiers by mapping each label to a unique output token and performing a single forward pass with structured prompts. The method combines class-to-token mappings, structured prompting, and LoRA-based fine-tuning to achieve strong zero-shot and few-shot performance and fast convergence, while offering an extension to continuous values via discretization. Across GLUE subsets and domain-specific benchmarks (e.g., MedNLI, MedMCQA, HateXplain), SALSA achieves state-of-the-art results and demonstrates robustness and scalability for high-throughput, real-world classification tasks. The approach reduces inference latency and preserves the base model through parameter-efficient adapters, making it well-suited for resource-constrained settings and broad applicability in NLP classification tasks.

Abstract

Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output token, and prompts are constructed to elicit a single-token response. During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.
Paper Structure (16 sections, 3 equations, 4 figures, 5 tables)

This paper contains 16 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Experiments and Results
  5. Datasets
  6. Analysis
  7. Conclusion
  8. Limitations
  9. Appendices
  10. Datasets
  11. Detailed Inference and Tuning Algorithm
  12. Training Details
  13. Prompt Construction Example
  14. Ablation Study on Label Mapping Strategies
  15. Discrete to Continuous Extension
  16. ...and 1 more sections

Figures (4)

  • Figure 1: SALSA single-token classification pipeline: each category is mapped to a distinct token, and the LLM’s logits determine the predicted label in one forward pass.
  • Figure 2: Convergence comparison between SALSA and Vanilla fine-tuning on RTE rte. SALSA achieves faster convergence with higher accuracy on both training and validation sets, indicating better generalization and training efficiency.
  • Figure 3: A compiled prompt from RTE dataset before applying a forward pass.
  • Figure 4: SALSA two-token classification pipeline: the LLM’s logits are used in a single pass to predict both the article’s topic (1–4) and its source (AI=1 or Human=2).