Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages

Tara Bogavelli, Oluwanifemi Bamgbose, Gabrielle Gauthier Melançon, Fanny Riols, Roshnee Sharma

TL;DR

The paper tackles enterprise-scale robustness of large language models by introducing a unified benchmark that stresses models with realistic perturbations across language, format, and instruction structure.It evaluates 11 models (4B to 120B+) on four enterprise tasks (case/chat summarization, Q&A, and entity slot-filling) under five perturbation categories, using content similarity and task-specific quality metrics to quantify robustness.Key findings show a non-linear relationship between model size and robustness, with training methodology and architecture sometimes outperforming larger models, and significant vulnerabilities to multilingual and formatting perturbations that can impact enterprise use cases.The work provides practical guidance for model selection and system design in real-world deployments, emphasizing robustness testing for multilingual, cross-lingual, and structured-output tasks and cautioning against relying solely on parameter count as a robustness proxy.

Abstract

Enterprise LLM applications require consistently high quality and reliable performance across diverse scenarios, demanding robustness to minor variations. Existing research shows that even small prompt changes can lead to substantial differences in output, but has mainly focused on a narrow set of perturbations with small academic datasets, limiting their relevance to real-world applications. To address this, we present a comprehensive benchmark suite that evaluates robustness across multiple perturbation types, including general text edits (e.g., punctuation, whitespace), formatting changes (e.g., JSON, YAML), multilingual and cross-lingual inputs, and positional variations in instructions. Evaluating 11 models ranging from 4B to 120B+ parameters, we find that minor perturbations reduce performance by up to 40 percentage points on key enterprise metrics. Critically, we demonstrate that the relationship between model size and robustness is more nuanced than conventional assumptions suggest: an 8B parameter model (Ministral 3 8B) outperforms most larger models, while another 8B model (Llama 3.1 8B) performs worst overall.

Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages

TL;DR

The paper tackles enterprise-scale robustness of large language models by introducing a unified benchmark that stresses models with realistic perturbations across language, format, and instruction structure.It evaluates 11 models (4B to 120B+) on four enterprise tasks (case/chat summarization, Q&A, and entity slot-filling) under five perturbation categories, using content similarity and task-specific quality metrics to quantify robustness.Key findings show a non-linear relationship between model size and robustness, with training methodology and architecture sometimes outperforming larger models, and significant vulnerabilities to multilingual and formatting perturbations that can impact enterprise use cases.The work provides practical guidance for model selection and system design in real-world deployments, emphasizing robustness testing for multilingual, cross-lingual, and structured-output tasks and cautioning against relying solely on parameter count as a robustness proxy.

Abstract

Enterprise LLM applications require consistently high quality and reliable performance across diverse scenarios, demanding robustness to minor variations. Existing research shows that even small prompt changes can lead to substantial differences in output, but has mainly focused on a narrow set of perturbations with small academic datasets, limiting their relevance to real-world applications. To address this, we present a comprehensive benchmark suite that evaluates robustness across multiple perturbation types, including general text edits (e.g., punctuation, whitespace), formatting changes (e.g., JSON, YAML), multilingual and cross-lingual inputs, and positional variations in instructions. Evaluating 11 models ranging from 4B to 120B+ parameters, we find that minor perturbations reduce performance by up to 40 percentage points on key enterprise metrics. Critically, we demonstrate that the relationship between model size and robustness is more nuanced than conventional assumptions suggest: an 8B parameter model (Ministral 3 8B) outperforms most larger models, while another 8B model (Llama 3.1 8B) performs worst overall.
Paper Structure (23 sections, 3 equations, 2 figures, 10 tables)

This paper contains 23 sections, 3 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Dataset Creation
  5. Perturbation Types
  6. Evaluation Metrics
  7. Content Similarity
  8. Quality Metrics
  9. Overall Score
  10. Experiments
  11. Models
  12. Overall Results Analysis
  13. Model Size Analysis
  14. Perturbation Category Analysis
  15. Universal vs. Model-Specific Difficulty
  16. ...and 8 more sections

Figures (2)

  • Figure 1: JSON Format Perturbation on a Chat Summarization Task The model performs a chat summarization task under two prompt conditions: standard (natural language) and perturbed (structured JSON format). We evaluate the outputs by measuring the quality (e.g., completeness) of each summary and assess robustness based on the consistency of model behavior across the two prompts.
  • Figure 2: Overall Scores on positional perturbations by model (mean ± std). The GPT 5 family demonstrates consistent advantages with low variance. Note the extremely high variance for GPT OSS 20B (±18.93) and GPT OSS 120B (±20.91), indicating unstable positional handling. Smaller open-source models (Llama 3.1 8B, Gemma 3 4B) show significant sensitivity to instruction ordering.