Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI

Fadel Mamar Seydou, Arnab Sharma

Abstract

In recent years, machine learning (ML) based software systems are increasingly deployed in several critical applications, yet systematic testing of their behavior remains challenging due to complex model architectures, large input spaces, and evolving deployment environments. Existing testing approaches often rely on generating test cases based on given requirements, which often fail to reveal critical bugs of modern ML models due to their complex nature. Most importantly, such approaches, although they can be used to detect the presence of specific failures in the ML software, they hardly provide any message as to how to fix such errors. To tackle this, in this paper, we present DeepFix, a tool for automated testing of the entire ML pipeline using an agentic AI framework. Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report. This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts and most importantly, also provides possible ways to fix these bugs. Additionally, DeepFix supports several types of ML software systems and can be integrated easily to any ML workflow, enabling continuous testing throughout the development lifecycle. We discuss our already validated cases as well as some planned validations designed to demonstrate how the agentic testing process can reveal hidden failure modes that remain undetected by conventional testing methods. A 5-minute screencast demonstrating the tool's core functionality is available at https://youtu.be/WfwZmFcQgBQ.

DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI

Abstract

In recent years, machine learning (ML) based software systems are increasingly deployed in several critical applications, yet systematic testing of their behavior remains challenging due to complex model architectures, large input spaces, and evolving deployment environments. Existing testing approaches often rely on generating test cases based on given requirements, which often fail to reveal critical bugs of modern ML models due to their complex nature. Most importantly, such approaches, although they can be used to detect the presence of specific failures in the ML software, they hardly provide any message as to how to fix such errors. To tackle this, in this paper, we present DeepFix, a tool for automated testing of the entire ML pipeline using an agentic AI framework. Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report. This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts and most importantly, also provides possible ways to fix these bugs. Additionally, DeepFix supports several types of ML software systems and can be integrated easily to any ML workflow, enabling continuous testing throughout the development lifecycle. We discuss our already validated cases as well as some planned validations designed to demonstrate how the agentic testing process can reveal hidden failure modes that remain undetected by conventional testing methods. A 5-minute screencast demonstrating the tool's core functionality is available at https://youtu.be/WfwZmFcQgBQ.
Paper Structure (9 sections, 2 figures, 2 tables)

This paper contains 9 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. DeepFix testing framework
  3. The testing approach
  4. Phase 1: ML Workflow Testing (client-side)
  5. Phase 2: Multi-agent analysis (server-side)
  6. Extension Points
  7. Validation
  8. Limitations
  9. Conclusion

Figures (2)

  • Figure 1: High-level architecture of DeepFix. The SDK (client) performs testing, stores and filters the unnecessary details, and sends them to the server via a POST request. The stateless server dispatches them to three parallel analysis agents, whose findings are synthesized by a cross-artifact reasoning agent backed by an LLM provider and a knowledge base of best practices.
  • Figure 2: The DeepFix Cross-Artifact Reasoning Procedure (Phase 2). Artifacts generated by parallel analyzers are aggregated, hypothesis formulated using the knowledge-base, and finalized through self-consistency decoding to output actionable recommendations.