Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures

TL;DR

This work investigates whether Small Language Models can reliably reverse-engineer ML pipeline structures from code, addressing the diversity and rapid evolution of the ML ecosystem. It presents a confirmatory study that benchmarks open-code–trained SLMs against two reference pipeline-studies (DS-Pipelines and DASWOW) using Cochran's Q, McNemar, and Pearson's chi-squared tests to assess accuracy and practice-pattern insights. Key contributions include identifying a best-performing SLM (SLMbest) that outperforms rule-based and supervised baselines, analyzing how taxonomy wording affects results, and evaluating cross-study goodness-of-fit for data-scientist practices. The approach includes a unified taxonomy Tunified, an execution plan for robust evaluation, and an exploration platform to visualize ML-practice patterns, emphasizing reproducibility and practical applicability in ML pipeline analysis.

Abstract

Background: Extracting the stages that structure Machine Learning (ML) pipelines from source code is key for gaining a deeper understanding of data science practices. However, the diversity caused by the constant evolution of the ML ecosystem (e.g., algorithms, libraries, datasets) makes this task challenging. Existing approaches either depend on non-scalable, manual labeling, or on ML classifiers that do not properly support the diversity of the domain. These limitations highlight the need for more flexible and reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and subsequently how they can advance our understanding of data science practices. Method: We conduct a confirmatory study based on two reference works selected for their relevance regarding current state-of-the-art's limitations. First, we compare several SLMs using Cochran's Q test. The best-performing model is then evaluated against the reference studies using two distinct McNemar's tests. We further analyze how variations in taxonomy definitions affect performance through an additional Cochran's Q test. Finally, a goodness-of-fit analysis is conducted using Pearson's chi-squared tests to compare our insights on data science practices with those from prior studies.