Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flow with FlorDB: Incremental Context Maintenance for the Machine Learning Lifecycle

Rolando Garcia, Pragya Kallanagoudar, Chithra Anand, Sarah E. Chasins, Joseph M. Hellerstein, Erin Michelle Turner Kerrison, Aditya G. Parameswaran

TL;DR

This paper shows how the component techniques come together to resolve classic software engineering trade-offs between agility and discipline and demonstrates how the data context framework covers a range of both ad-hoc metadata as well as special cases treated today by bespoke feature stores and model repositories.

Abstract

In this paper we present techniques to incrementally harvest and query arbitrary metadata from machine learning pipelines, without disrupting agile practices. We center our approach on the developer-favored technique for generating metadata -- log statements -- leveraging the fact that logging creates context. We show how hindsight logging allows such statements to be added and executed post-hoc, without requiring developer foresight. Relational views of incomplete metadata can be queried to dynamically materialize new metadata in bulk and on demand across multiple versions of workflows. This is done in a "metadata later" style, off the critical path of agile development. We realize these ideas in a system called FlorDB and demonstrate how the data context framework covers a range of both ad-hoc metadata as well as special cases treated today by bespoke feature stores and model repositories. Through a usage scenario -- including both ML and human feedback -- we illustrate how the component techniques come together to resolve classic software engineering trade-offs between agility and discipline.

Flow with FlorDB: Incremental Context Maintenance for the Machine Learning Lifecycle

TL;DR

This paper shows how the component techniques come together to resolve classic software engineering trade-offs between agility and discipline and demonstrates how the data context framework covers a range of both ad-hoc metadata as well as special cases treated today by bespoke feature stores and model repositories.

Abstract

In this paper we present techniques to incrementally harvest and query arbitrary metadata from machine learning pipelines, without disrupting agile practices. We center our approach on the developer-favored technique for generating metadata -- log statements -- leveraging the fact that logging creates context. We show how hindsight logging allows such statements to be added and executed post-hoc, without requiring developer foresight. Relational views of incomplete metadata can be queried to dynamically materialize new metadata in bulk and on demand across multiple versions of workflows. This is done in a "metadata later" style, off the critical path of agile development. We realize these ideas in a system called FlorDB and demonstrate how the data context framework covers a range of both ad-hoc metadata as well as special cases treated today by bespoke feature stores and model repositories. Through a usage scenario -- including both ML and human feedback -- we illustrate how the component techniques come together to resolve classic software engineering trade-offs between agility and discipline.
Paper Structure (17 sections, 6 figures)

This paper contains 17 sections, 6 figures.

Table of Contents

  1. Introduction
  2. A Crisis of Metadata Management
  3. Goals and Contributions
  4. Multiversion Hindsight Logging
  5. FlorDB Extended API
  6. Incremental Context Maintenance
  7. Application Context
  8. Behavioral & Change Context
  9. PDF Parser Demo
  10. PDF Extraction & Text Featurization
  11. Inference Pipeline
  12. Training Pipeline
  13. Closing the Loop: Feedback via UI
  14. Discussion
  15. Implications for Social Justice Research
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Extended FlorDB data model in Crow's Foot notation. Basic tables denoted in white; virtual tables in gray.
  • Figure 2: ML Pipeline with Feedback: Makefile, Dataflow Diagram, and Flor Dataframe.
  • Figure 3: Data featurization with FlorDB
  • Figure 4: Screenshot of the PDF Parser (left) and its respective Makefile (right).
  • Figure 5: Training on labeled data managed by FlorDB
  • ...and 1 more figures