Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs

Guangba Yu, Zirui Wang, Yujie Huang, Renyi Zhong, Yuedong Zhong, Yilun Wang, Michael R. Lyu

TL;DR

This paper addresses the reliability challenges of open-source LLMs deployed on user-managed infrastructure, focusing on the First Mile between fine-tuning and inference. It introduces a ground-truth, manually labeled dataset of 705 user-reported failures from DeepSeek, Llama, and Qwen, along with two hierarchical taxonomies for symptoms and root causes. The study reveals a paradigm shift: failures are dominated by deployment-stack fragility (environment, dependencies, and configuration) and by semantic-generation anomalies, rather than intrinsic model defects, with clear diagnostic heuristics and cross-series generalizability. The findings inform practical guidance for practitioners and providers, including standardized deployment stacks, stage-aware tooling, and behavioral validation for generation quality, underscoring the need for ecosystem-level tooling and documentation to improve LLM reliability in real-world use.

Abstract

The democratization of open-source Large Language Models (LLMs) allows users to fine-tune and deploy models on local infrastructure but exposes them to a First Mile deployment landscape. Unlike black-box API consumption, the reliability of user-managed orchestration remains a critical blind spot. To bridge this gap, we conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems. Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack. We identify three key phenomena: (1) Diagnostic Divergence: runtime crashes distinctively signal infrastructure friction, whereas incorrect functionality serves as a signature for internal tokenizer defects. (2) Systemic Homogeneity: Root causes converge across divergent series, confirming reliability barriers are inherent to the shared ecosystem rather than specific architectures. (3) Lifecycle Escalation: Barriers escalate from intrinsic configuration struggles during fine-tuning to compounded environmental incompatibilities during inference. Supported by our publicly available dataset, these insights provide actionable guidance for enhancing the reliability of the LLM landscape.

Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs

TL;DR

This paper addresses the reliability challenges of open-source LLMs deployed on user-managed infrastructure, focusing on the First Mile between fine-tuning and inference. It introduces a ground-truth, manually labeled dataset of 705 user-reported failures from DeepSeek, Llama, and Qwen, along with two hierarchical taxonomies for symptoms and root causes. The study reveals a paradigm shift: failures are dominated by deployment-stack fragility (environment, dependencies, and configuration) and by semantic-generation anomalies, rather than intrinsic model defects, with clear diagnostic heuristics and cross-series generalizability. The findings inform practical guidance for practitioners and providers, including standardized deployment stacks, stage-aware tooling, and behavioral validation for generation quality, underscoring the need for ecosystem-level tooling and documentation to improve LLM reliability in real-world use.

Abstract

The democratization of open-source Large Language Models (LLMs) allows users to fine-tune and deploy models on local infrastructure but exposes them to a First Mile deployment landscape. Unlike black-box API consumption, the reliability of user-managed orchestration remains a critical blind spot. To bridge this gap, we conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems. Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack. We identify three key phenomena: (1) Diagnostic Divergence: runtime crashes distinctively signal infrastructure friction, whereas incorrect functionality serves as a signature for internal tokenizer defects. (2) Systemic Homogeneity: Root causes converge across divergent series, confirming reliability barriers are inherent to the shared ecosystem rather than specific architectures. (3) Lifecycle Escalation: Barriers escalate from intrinsic configuration struggles during fine-tuning to compounded environmental incompatibilities during inference. Supported by our publicly available dataset, these insights provide actionable guidance for enhancing the reliability of the LLM landscape.
Paper Structure (33 sections, 8 figures, 2 tables)

This paper contains 33 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Empirical Study Methodology
  3. Targeted Model Selection
  4. Failure-related Issue Collection
  5. Data Labeling
  6. Rationale for Manual Analysis
  7. Taxonomy Development
  8. Labeling Procedure and Quality Control
  9. Empirical Study Results
  10. RQ1: Symptoms of LLM Failures
  11. Runtime Crash [A]
  12. Process Hang [B]
  13. Incorrect Functionality [C]
  14. Poor Performance [D]
  15. Comparison with DL System in Symptoms
  16. ...and 18 more sections

Figures (8)

  • Figure 1: Overview of empirical study methodology.
  • Figure 2: Symptom taxonomy of failure in open-source LLMs.
  • Figure 3: Illustrative cases of various Runtime Crash failures reported in GitHub issues.
  • Figure 4: Illustrative cases of non-crashing failures reported in GitHub issues.
  • Figure 5: Root cause taxonomy of failures in open-source LLMs.
  • ...and 3 more figures