Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparative Efficiency Analysis of Lightweight Transformer Models: A Multi-Domain Empirical Benchmark for Enterprise NLP Deployment

Muhammad Shahmeer Khan

TL;DR

This work addresses the need for deployment-ready, lightweight Transformer models capable of handling multi-domain enterprise NLP tasks. It benchmarks DistilBERT, MiniLM, and ALBERT across three domains—IMDB sentiment, AG News topics, and Measuring Hate Speech moderation—using both accuracy-based and efficiency metrics. The study finds that no single model dominates all dimensions: ALBERT often achieves top task accuracy, MiniLM offers superior inference speed and throughput, and DistilBERT delivers consistent accuracy with competitive efficiency. The results yield a practical decision framework for enterprise deployment, recommending MiniLM for latency-sensitive pipelines, DistilBERT for balanced performance, and ALBERT for memory-constrained settings, all under a reproducible, fixed-finetuning regime that mirrors real-world constraints.

Abstract

In the rapidly evolving landscape of enterprise natural language processing (NLP), the demand for efficient, lightweight models capable of handling multi-domain text automation tasks has intensified. This study conducts a comparative analysis of three prominent lightweight Transformer models - DistilBERT, MiniLM, and ALBERT - across three distinct domains: customer sentiment classification, news topic classification, and toxicity and hate speech detection. Utilizing datasets from IMDB, AG News, and the Measuring Hate Speech corpus, we evaluated performance using accuracy-based metrics including accuracy, precision, recall, and F1-score, as well as efficiency metrics such as model size, inference time, throughput, and memory usage. Key findings reveal that no single model dominates all performance dimensions. ALBERT achieves the highest task-specific accuracy in multiple domains, MiniLM excels in inference speed and throughput, and DistilBERT demonstrates the most consistent accuracy across tasks while maintaining competitive efficiency. All results reflect controlled fine-tuning under fixed enterprise-oriented constraints rather than exhaustive hyperparameter optimization. These results highlight trade-offs between accuracy and efficiency, recommending MiniLM for latency-sensitive enterprise applications, DistilBERT for balanced performance, and ALBERT for resource-constrained environments.

Comparative Efficiency Analysis of Lightweight Transformer Models: A Multi-Domain Empirical Benchmark for Enterprise NLP Deployment

TL;DR

This work addresses the need for deployment-ready, lightweight Transformer models capable of handling multi-domain enterprise NLP tasks. It benchmarks DistilBERT, MiniLM, and ALBERT across three domains—IMDB sentiment, AG News topics, and Measuring Hate Speech moderation—using both accuracy-based and efficiency metrics. The study finds that no single model dominates all dimensions: ALBERT often achieves top task accuracy, MiniLM offers superior inference speed and throughput, and DistilBERT delivers consistent accuracy with competitive efficiency. The results yield a practical decision framework for enterprise deployment, recommending MiniLM for latency-sensitive pipelines, DistilBERT for balanced performance, and ALBERT for memory-constrained settings, all under a reproducible, fixed-finetuning regime that mirrors real-world constraints.

Abstract

In the rapidly evolving landscape of enterprise natural language processing (NLP), the demand for efficient, lightweight models capable of handling multi-domain text automation tasks has intensified. This study conducts a comparative analysis of three prominent lightweight Transformer models - DistilBERT, MiniLM, and ALBERT - across three distinct domains: customer sentiment classification, news topic classification, and toxicity and hate speech detection. Utilizing datasets from IMDB, AG News, and the Measuring Hate Speech corpus, we evaluated performance using accuracy-based metrics including accuracy, precision, recall, and F1-score, as well as efficiency metrics such as model size, inference time, throughput, and memory usage. Key findings reveal that no single model dominates all performance dimensions. ALBERT achieves the highest task-specific accuracy in multiple domains, MiniLM excels in inference speed and throughput, and DistilBERT demonstrates the most consistent accuracy across tasks while maintaining competitive efficiency. All results reflect controlled fine-tuning under fixed enterprise-oriented constraints rather than exhaustive hyperparameter optimization. These results highlight trade-offs between accuracy and efficiency, recommending MiniLM for latency-sensitive enterprise applications, DistilBERT for balanced performance, and ALBERT for resource-constrained environments.
Paper Structure (26 sections, 6 figures, 4 tables)

This paper contains 26 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformer Models in Enterprise NLP
  4. Lightweight Transformer Architectures
  5. Multi-Domain NLP Evaluation
  6. Methodology
  7. Models
  8. Datasets and Tasks
  9. Experimental Setup
  10. Evaluation Protocol
  11. Accuracy-based metrics.
  12. Efficiency metrics.
  13. Efficiency measurement details.
  14. Reproducibility and reporting.
  15. Results
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Comparison of inference latency, throughput, and model size across models for Domain 1 (IMDB Sentiment).
  • Figure 2: Comparison of inference latency, throughput, and model size across models for Domain 2 (AG News).
  • Figure 3: Comparison of inference latency, throughput, and model size across models for Domain 3 (Cyberbullying).
  • Figure 4: Confusion matrices for Domain 1 (IMDB Sentiment Classification) using DistilBERT, MiniLM, and ALBERT.
  • Figure 5: Confusion matrices for Domain 2 (AG News Topic Classification) using DistilBERT, MiniLM, and ALBERT.
  • ...and 1 more figures