Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

Yunfan Zhang, Yijie Bei, Jetashree Ravi, Pawel Garbacki

TL;DR

This work evaluates 11 LLMs and presents key findings on their instruction following behavior in enterprise scenarios, and introduces FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns.

Abstract

Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

TL;DR

This work evaluates 11 LLMs and presents key findings on their instruction following behavior in enterprise scenarios, and introduces FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns.

Abstract

Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.
Paper Structure (36 sections, 2 figures, 2 tables)

This paper contains 36 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction Following Benchmarks for LLMs
  4. LLM Benchmarks for Enterprise Settings
  5. Methodology
  6. Output Format Compliance
  7. Rationale
  8. Task Design
  9. Ordered Responses
  10. Rationale
  11. Task Design
  12. Item Ranking
  13. Rationale
  14. Task Design
  15. Overconfidence
  16. ...and 21 more sections

Figures (2)

  • Figure 1: Overview of FireBench. The benchmark evaluates six instruction-following capabilities critical to enterprise and API settings, each grounded in real-world application domains.
  • Figure 2: (a) Overall score and standard deviation across categories for the top-6 models by overall performance. All models exhibit high variance, indicating inconsistent instruction-following ability across categories. (b) Relative ranking of the same six models within each category. Model rankings shift substantially from one category to another, demonstrating that no single model dominates uniformly.