Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Grammar of Machine Learning Workflows

Simon Roth

TL;DR

A grammar is proposed that decomposes the supervised learning lifecycle into 7 kernel primitives connected by a typed directed acyclic graph (DAG), with four hard constraints that reject the two most damaging leakage classes at call time.

Abstract

Data leakage affected 294 published papers across 17 scientific fields (Kapoor & Narayanan, 2023). The dominant response has been documentation: checklists, linters, best-practice guides. Documentation does not prevent these failures. This paper proposes a structural remedy: a grammar that decomposes the supervised learning lifecycle into 7 kernel primitives connected by a typed directed acyclic graph (DAG), with four hard constraints that reject the two most damaging leakage classes at call time. The grammar's core contribution is the terminal assess constraint: a runtime-enforced evaluate/assess boundary where repeated test-set assessment is rejected by a guard on a nominally distinct Evidence type. A companion study across 2,047 experimental instances quantifies why this matters: selection leakage inflates performance by d_z = 0.93 and memorization leakage by d_z = 0.53-1.11. Three separate implementations (Python, R, and Julia) confirm the claims. The appendix specification lets anyone build a conforming version.

A Grammar of Machine Learning Workflows

TL;DR

A grammar is proposed that decomposes the supervised learning lifecycle into 7 kernel primitives connected by a typed directed acyclic graph (DAG), with four hard constraints that reject the two most damaging leakage classes at call time.

Abstract

Data leakage affected 294 published papers across 17 scientific fields (Kapoor & Narayanan, 2023). The dominant response has been documentation: checklists, linters, best-practice guides. Documentation does not prevent these failures. This paper proposes a structural remedy: a grammar that decomposes the supervised learning lifecycle into 7 kernel primitives connected by a typed directed acyclic graph (DAG), with four hard constraints that reject the two most damaging leakage classes at call time. The grammar's core contribution is the terminal assess constraint: a runtime-enforced evaluate/assess boundary where repeated test-set assessment is rejected by a guard on a nominally distinct Evidence type. A companion study across 2,047 experimental instances quantifies why this matters: selection leakage inflates performance by d_z = 0.93 and memorization leakage by d_z = 0.53-1.11. Three separate implementations (Python, R, and Julia) confirm the claims. The appendix specification lets anyone build a conforming version.
Paper Structure (14 sections, 1 figure)

This paper contains 14 sections, 1 figure.

Table of Contents

  1. Introduction
  2. A Motivating Example
  3. The Grammar
  4. Kernel primitives
  5. The type DAG
  6. The four hard constraints
  7. Strategy families
  8. The terminal boundary
  9. Design Properties
  10. Reachability
  11. Generativity
  12. Rejection criterion
  13. Embedding in Code
  14. Triple implementation

Figures (1)

  • Figure 1: The ML workflow grammar as a typed DAG. Diamond: split decision. Solid arrows: permitted data flow. Dashed arrows: locked until terminal assessment. Gate ($\times$): test partition held until assess is called once, producing terminal evidence. The fit--evaluate loop (grayed out) iterates freely on train/valid partitions; assess commits on the held-out test partition exactly once.