Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing Quality Challenges in Deep Learning: The Role of MLOps and Domain Knowledge

Santiago del Rey, Adrià Medina, Xavier Franch, Silverio Martínez-Fernández

TL;DR

Addressing quality challenges in DL, the paper investigates how MLOps and domain knowledge influence QA attributes in an image-recognition DL system for live chess broadcasting. It demonstrates practical integration of MLflow-based experiment tracking and energy monitoring, and compares domain-free versus domain-aware algorithms (IA, CPA, CPS, TK-k). The results show substantial gains in accuracy (up to 96.85%), reduced latency, and lower energy consumption when applying domain knowledge, along with clear guidelines on when to halt further optimization. The findings offer actionable insights for practitioners to operationalize MLOps and domain knowledge to improve reliability and efficiency in edge DL deployments.

Abstract

Deep learning (DL) systems present unique challenges in software engineering, especially concerning quality attributes like correctness and resource efficiency. While DL models excel in specific tasks, engineering DL systems is still essential. The effort, cost, and potential diminishing returns of continual improvements must be carefully evaluated, as software engineers often face the critical decision of when to stop refining a system relative to its quality attributes. This experience paper explores the role of MLOps practices -- such as monitoring and experiment tracking -- in creating transparent and reproducible experimentation environments that enable teams to assess and justify the impact of design decisions on quality attributes. Furthermore, we report on experiences addressing the quality challenges by embedding domain knowledge into the design of a DL model and its integration within a larger system. The findings offer actionable insights into the benefits of domain knowledge and MLOps and the strategic consideration of when to limit further optimizations in DL projects to maximize overall system quality and reliability.

Addressing Quality Challenges in Deep Learning: The Role of MLOps and Domain Knowledge

TL;DR

Addressing quality challenges in DL, the paper investigates how MLOps and domain knowledge influence QA attributes in an image-recognition DL system for live chess broadcasting. It demonstrates practical integration of MLflow-based experiment tracking and energy monitoring, and compares domain-free versus domain-aware algorithms (IA, CPA, CPS, TK-k). The results show substantial gains in accuracy (up to 96.85%), reduced latency, and lower energy consumption when applying domain knowledge, along with clear guidelines on when to halt further optimization. The findings offer actionable insights for practitioners to operationalize MLOps and domain knowledge to improve reliability and efficiency in edge DL deployments.

Abstract

Deep learning (DL) systems present unique challenges in software engineering, especially concerning quality attributes like correctness and resource efficiency. While DL models excel in specific tasks, engineering DL systems is still essential. The effort, cost, and potential diminishing returns of continual improvements must be carefully evaluated, as software engineers often face the critical decision of when to stop refining a system relative to its quality attributes. This experience paper explores the role of MLOps practices -- such as monitoring and experiment tracking -- in creating transparent and reproducible experimentation environments that enable teams to assess and justify the impact of design decisions on quality attributes. Furthermore, we report on experiences addressing the quality challenges by embedding domain knowledge into the design of a DL model and its integration within a larger system. The findings offer actionable insights into the benefits of domain knowledge and MLOps and the strategic consideration of when to limit further optimizations in DL projects to maximize overall system quality and reliability.
Paper Structure (21 sections, 1 figure, 1 table)

This paper contains 21 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Study design
  4. Research Goal and questions
  5. The DL system for image recognition
  6. Data analysis
  7. Applying traceability and reproducibility with MLOps (RQ1)
  8. Experiment tracking from new data collected in operation
  9. Energy monitoring
  10. Automatic data collection and processing
  11. Incorporating Domain Knowledge (RQ2)
  12. From domain-free to domain-aware algorithms
  13. Experimental setting and execution
  14. The Impact of Domain Knowledge
  15. Accuracy (RQ2.1)
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Software development life cycle of the DL system.