Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenges of Using Pre-trained Models: the Practitioners' Perspective

Xin Tan, Taichuan Li, Ruohe Chen, Fang Liu, Li Zhang

TL;DR

The paper investigates the practical challenges of pre-trained models (PTMs) from a practitioner's lens by mining Stack Overflow questions (5,896 total; 430 manually analyzed) to build a 42-code, three-category taxonomy of PTM challenges. It shows PTM-related questions are rising in popularity but are relatively harder to answer, with higher %no-acc, longer response times, and lower answer-to-view ratios compared to non-PTM topics. The taxonomy highlights dominant issues in Model Lifecycle Management, as well as unique PTM-specific challenges like fine-tuning, output understanding, and memory management, offering actionable guidance for researchers, practitioners, vendors, and educators. The work provides a dataset and structured insights to improve tooling, documentation, education, and cross-platform compatibility in PTMs, with practical impact for accelerating adoption and responsible deployment.

Abstract

The challenges associated with using pre-trained models (PTMs) have not been specifically investigated, which hampers their effective utilization. To address this knowledge gap, we collected and analyzed a dataset of 5,896 PTM-related questions on Stack Overflow. We first analyze the popularity and difficulty trends of PTM-related questions. We find that PTM-related questions are becoming more and more popular over time. However, it is noteworthy that PTM-related questions not only have a lower response rate but also exhibit a longer response time compared to many well-researched topics in software engineering. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs. To delve into the specific challenges, we manually annotate 430 PTM-related questions, categorizing them into a hierarchical taxonomy of 42 codes (i.e., leaf nodes) and three categories. This taxonomy encompasses many PTM prominent challenges such as fine-tuning, output understanding, and prompt customization, which reflects the gaps between current techniques and practical needs. We discuss the implications of our study for PTM practitioners, vendors, and educators, and suggest possible directions and solutions for future research.

Challenges of Using Pre-trained Models: the Practitioners' Perspective

TL;DR

The paper investigates the practical challenges of pre-trained models (PTMs) from a practitioner's lens by mining Stack Overflow questions (5,896 total; 430 manually analyzed) to build a 42-code, three-category taxonomy of PTM challenges. It shows PTM-related questions are rising in popularity but are relatively harder to answer, with higher %no-acc, longer response times, and lower answer-to-view ratios compared to non-PTM topics. The taxonomy highlights dominant issues in Model Lifecycle Management, as well as unique PTM-specific challenges like fine-tuning, output understanding, and memory management, offering actionable guidance for researchers, practitioners, vendors, and educators. The work provides a dataset and structured insights to improve tooling, documentation, education, and cross-platform compatibility in PTMs, with practical impact for accelerating adoption and responsible deployment.

Abstract

The challenges associated with using pre-trained models (PTMs) have not been specifically investigated, which hampers their effective utilization. To address this knowledge gap, we collected and analyzed a dataset of 5,896 PTM-related questions on Stack Overflow. We first analyze the popularity and difficulty trends of PTM-related questions. We find that PTM-related questions are becoming more and more popular over time. However, it is noteworthy that PTM-related questions not only have a lower response rate but also exhibit a longer response time compared to many well-researched topics in software engineering. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs. To delve into the specific challenges, we manually annotate 430 PTM-related questions, categorizing them into a hierarchical taxonomy of 42 codes (i.e., leaf nodes) and three categories. This taxonomy encompasses many PTM prominent challenges such as fine-tuning, output understanding, and prompt customization, which reflects the gaps between current techniques and practical needs. We discuss the implications of our study for PTM practitioners, vendors, and educators, and suggest possible directions and solutions for future research.
Paper Structure (31 sections, 3 figures, 2 tables)

This paper contains 31 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. PTMs
  4. Pre-training Toolkits
  5. Data Preparation
  6. RQ1: Popularity Trend
  7. Motivation
  8. Approach
  9. Results
  10. RQ2: Difficulty
  11. Motivation
  12. Approach
  13. Results
  14. RQ3: Taxonomy of Challenges
  15. Motivation
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Popularity Trend of PTM-Related Questions and Top Six Tags
  • Figure 2: Time Needed to Receive an Accepted Answer
  • Figure 3: Taxonomy of PTM Challenges. The challenges highlighted in codes indicate that these challenges are also prevalent in the broader field of deep learning zhao2021state, while others are prevalent in PTMs.