Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ensuring Robustness in ML-enabled Software Systems: A User Survey

Hala Abdelkader, Mohamed Abdelrazek, Priya Rani, Rajesh Vasa, Jean-Guy Schneider

TL;DR

This paper tackles robustness in ML-enabled software by addressing silent failures, OOD data, and adversarial threats that threaten production reliability. It introduces the ML-On-Rails protocol, a unified framework that combines input validation, OOD and adversarial detection, explainability via SHAP, and a model-to-software communication layer based on HTTP status codes. A practitioner survey validates the approach, revealing practical challenges, gaps in current tooling, and broad support for standardized safeguards, while also guiding refinements to improve usability and effectiveness. The study highlights the need for better documentation, data drift monitoring, ensemble safety strategies, and LLMS-specific safeguards to enhance production trustworthiness. Collectively, the work provides a concrete, actionable blueprint for engineers to improve robustness and transparency in ML-enabled systems, with clear directions for future enhancements and adoption in real-world contexts.

Abstract

Ensuring robustness in ML-enabled software systems requires addressing critical challenges, such as silent failures, out-of-distribution (OOD) data, and adversarial attacks. Traditional software engineering practices, which rely on predefined logic, are insufficient for ML components that depend on data and probabilistic decision-making. To address these challenges, we propose the ML-On-Rails protocol, a unified framework designed to enhance the robustness and trustworthiness of ML-enabled systems in production. This protocol integrates key safeguards such as OOD detection, adversarial attack detection, input validation, and explainability. It also includes a model-to-software communication framework using HTTP status codes to enhance transparency in reporting model outcomes and errors. To align our approach with real-world challenges, we conducted a practitioner survey, which revealed major robustness issues, gaps in current solutions, and highlighted how a standardised protocol such as ML-On-Rails can improve system robustness. Our findings highlight the need for more support and resources for engineers working with ML systems. Finally, we outline future directions for refining the proposed protocol, leveraging insights from the survey and real-world applications to continually enhance its effectiveness.

Ensuring Robustness in ML-enabled Software Systems: A User Survey

TL;DR

This paper tackles robustness in ML-enabled software by addressing silent failures, OOD data, and adversarial threats that threaten production reliability. It introduces the ML-On-Rails protocol, a unified framework that combines input validation, OOD and adversarial detection, explainability via SHAP, and a model-to-software communication layer based on HTTP status codes. A practitioner survey validates the approach, revealing practical challenges, gaps in current tooling, and broad support for standardized safeguards, while also guiding refinements to improve usability and effectiveness. The study highlights the need for better documentation, data drift monitoring, ensemble safety strategies, and LLMS-specific safeguards to enhance production trustworthiness. Collectively, the work provides a concrete, actionable blueprint for engineers to improve robustness and transparency in ML-enabled systems, with clear directions for future enhancements and adoption in real-world contexts.

Abstract

Ensuring robustness in ML-enabled software systems requires addressing critical challenges, such as silent failures, out-of-distribution (OOD) data, and adversarial attacks. Traditional software engineering practices, which rely on predefined logic, are insufficient for ML components that depend on data and probabilistic decision-making. To address these challenges, we propose the ML-On-Rails protocol, a unified framework designed to enhance the robustness and trustworthiness of ML-enabled systems in production. This protocol integrates key safeguards such as OOD detection, adversarial attack detection, input validation, and explainability. It also includes a model-to-software communication framework using HTTP status codes to enhance transparency in reporting model outcomes and errors. To align our approach with real-world challenges, we conducted a practitioner survey, which revealed major robustness issues, gaps in current solutions, and highlighted how a standardised protocol such as ML-On-Rails can improve system robustness. Our findings highlight the need for more support and resources for engineers working with ML systems. Finally, we outline future directions for refining the proposed protocol, leveraging insights from the survey and real-world applications to continually enhance its effectiveness.
Paper Structure (17 sections, 10 figures, 3 tables)

This paper contains 17 sections, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. ML-On-Rails protocol
  3. Input validation Safeguard
  4. Model Explainability Safeguard
  5. Out-of-Distribution Data Detection Safeguard
  6. Adversarial Attacks Detection Safeguard
  7. Model-2-Software Communication Framework
  8. Practitioner survey
  9. Recruitment of Participants
  10. Results
  11. Discussion
  12. Challenges Software Engineers Face with ML Models
  13. Identifying ML Model Issues in Production
  14. Protocol Design
  15. ML Protocol Evaluation
  16. ...and 2 more sections

Figures (10)

  • Figure 1: An illustrative example of the proposed ML-On-Rails protocol that shows the sequence of operations of a given input and possible outputs
  • Figure 2: Participant roles within the organization
  • Figure 3: Distribution of application types among participants
  • Figure 4: Responses on challenges with machine learning terminology
  • Figure 5: Process for identifying ML model performance issues in production
  • ...and 5 more figures