TorchQL: A Programming Framework for Integrity Constraints in Machine Learning

TL;DR

This work presents TorchQL, a programming framework to evaluate and improve the correctness of machine learning applications, which seamlessly integrates relational algebra with functional programming to allow for highly expressive queries using only eight intuitive operators.

Abstract

Finding errors in machine learning applications requires a thorough exploration of their behavior over data. Existing approaches used by practitioners are often ad-hoc and lack the abstractions needed to scale this process. We present TorchQL, a programming framework to evaluate and improve the correctness of machine learning applications. TorchQL allows users to write queries to specify and check integrity constraints over machine learning models and datasets. It seamlessly integrates relational algebra with functional programming to allow for highly expressive queries using only eight intuitive operators. We evaluate TorchQL on diverse use-cases including finding critical temporal inconsistencies in objects detected across video frames in autonomous driving, finding data imputation errors in time-series medical records, finding data labeling errors in real-world images, and evaluating biases and constraining outputs of language models. Our experiments show that TorchQL enables up to 13x faster query executions than baselines like Pandas and MongoDB, and up to 40% shorter queries than native Python. We also conduct a user study and find that TorchQL is natural enough for developers familiar with Python to specify complex integrity constraints.

Figures (24)

• Figure 1: (a) shows three consecutive frames in a video from the Cityscapes self-driving object-detection dataset. OneFormer, a state-of-the-art model, detects pedestrians in the center of the first two frames but not in the third. We formalize this fault as a violation of an integrity constraint. (b) shows a TorchQL query to find all such violations over a dataset named 'preds' consisting of video frames along with the model's predictions.
• Figure 2: Initializing a TorchQL database and running queries over it.
• Figure 3: Queries for finding individual frames in which at least one pedestrian is predicted.
• Figure 4: Queries for finding pairs of consecutive frames with different predicted pedestrian counts.
• Figure 5: Queries for finding sequences of three consecutive frames where a pedestrian is detected in the center of the first two frames but not in the third.