SemPipes -- Optimizable Semantic Data Operators for Tabular Machine Learning Pipelines

TL;DR

SemPipes tackles the complexity of preparing tabular data for machine learning by introducing declarative semantic data operators that are powered by LLMs during training to synthesize optimized operator implementations. The approach separates high-level pipeline structure from data-centric transformations, enabling automatic optimization of data operations via evolutionary search while ensuring efficient inference without LLM usage. Empirical results across diverse tasks show that semantic operators improve predictive performance and simplify pipelines for both expert-designed and agent-generated code, often matching or surpassing specialized feature engineering and data processing baselines. The work provides a practical, open-source framework for integrating LLM-driven data operations into real-world tabular ML workflows, with clear avenues for extending operator coverage and cost-aware optimization.

Abstract

Real-world machine learning on tabular data relies on complex data preparation pipelines for prediction, data integration, augmentation, and debugging. Designing these pipelines requires substantial domain expertise and engineering effort, motivating the question of how large language models (LLMs) can support tabular ML through code synthesis. We introduce SemPipes, a novel declarative programming model that integrates LLM-powered semantic data operators into tabular ML pipelines. Semantic operators specify data transformations in natural language while delegating execution to a runtime system. During training, SemPipes synthesizes custom operator implementations based on data characteristics, operator instructions, and pipeline context. This design enables the automatic optimization of data operations in a pipeline via LLM-based code synthesis guided by evolutionary search. We evaluate SemPipes across diverse tabular ML tasks and show that semantic operators substantially improve end-to-end predictive performance for both expert-designed and agent-generated pipelines, while reducing pipeline complexity. We implement SemPipes in Python and release it at https://github.com/deem-data/sempipes/tree/v1.

Figures (4)

• Figure 1: High-level overview of SemPipes: ML pipelines are defined in Python using standard data science libraries, including pandas for tabular data manipulation, numpy for numerical computation, scikit-learn for feature encoding, and models compatible with the scikit-learn ecosystem. SemPipes extends these pipelines with semantic data operators, which delegate selected data-centric operations to LLMs. During training, when a pipeline is "fitted" to the training data, SemPipessynthesizes custom implementations for the semantic operators based on data characteristics, natural language instructions, and pipeline context. The synthesized code may execute locally on the CPU or leverage a pretrained model on a local GPU. Once a pipeline is defined, the implementations of its semantic operators can be tuned on a validation set. SemPipes performs evolutionary search under a given search policy (e.g., Monte Carlo tree search), using downstream validation performance as the fitness function. During search, operator implementations are iteratively mutated via reflective prompting using prior performance scores and a tree-structured memory.
• Figure 2: Toy example of code synthesis for a semantic data operator. At training time, SemPipes synthesizes custom operator code conditioned on input data characteristics (e.g., size, column names, and statistics), the natural language instruction, and pipeline context such as the learning task and model type. The synthesized code is validated by executing it on a sample of the input data and by checking operator-specific output constraints.
• Figure 3: AUROC scores without feature engineering (No FE), and for automated feature engineering on three datasets with Random Forest (RF) and TabPFN.
• Figure 4: Optimization effectiveness of LLMs for code synthesis in semantic data operators. Search policies include Monte Carlo tree search (MCTS), truncation selection (TS), greedy tree search (GTS) or random search (RS).