Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from Uncertain Data: From Possible Worlds to Possible Models

Jiongli Zhu, Su Feng, Boris Glavic, Babak Salimi

TL;DR

The paper tackles learning under data uncertainty by embracing possible world semantics and over-approximating all data variations with zonotopes in an abstract interpretation framework. It develops Zorro, which symbolically executes gradient descent across all possible datasets to produce a fixed-point representation that soundly bounds all optimal linear models, with a closed-form solution for ridge regression under the abstract framework. The work introduces linearization and order-reduction techniques to control symbolic growth and proves a fixed-point existence under mild conditions, enabling efficient prediction-range certificates and robustness analysis. Empirically, Zorro demonstrates improved robustness Certification over baselines, supports causal-inference analysis with bound guarantees, and highlights practical guidance on regularization under uncertainty. Overall, the method provides a principled, tractable way to quantify and certify training-time uncertainty for linear models, with potential extensions to broader architectures and uncertainty modalities.

Abstract

We introduce an efficient method for learning linear models from uncertain data, where uncertainty is represented as a set of possible variations in the data, leading to predictive multiplicity. Our approach leverages abstract interpretation and zonotopes, a type of convex polytope, to compactly represent these dataset variations, enabling the symbolic execution of gradient descent on all possible worlds simultaneously. We develop techniques to ensure that this process converges to a fixed point and derive closed-form solutions for this fixed point. Our method provides sound over-approximations of all possible optimal models and viable prediction ranges. We demonstrate the effectiveness of our approach through theoretical and empirical analysis, highlighting its potential to reason about model and prediction uncertainty due to data quality issues in training data.

Learning from Uncertain Data: From Possible Worlds to Possible Models

TL;DR

The paper tackles learning under data uncertainty by embracing possible world semantics and over-approximating all data variations with zonotopes in an abstract interpretation framework. It develops Zorro, which symbolically executes gradient descent across all possible datasets to produce a fixed-point representation that soundly bounds all optimal linear models, with a closed-form solution for ridge regression under the abstract framework. The work introduces linearization and order-reduction techniques to control symbolic growth and proves a fixed-point existence under mild conditions, enabling efficient prediction-range certificates and robustness analysis. Empirically, Zorro demonstrates improved robustness Certification over baselines, supports causal-inference analysis with bound guarantees, and highlights practical guidance on regularization under uncertainty. Overall, the method provides a principled, tractable way to quantify and certify training-time uncertainty for linear models, with potential extensions to broader architectures and uncertainty modalities.

Abstract

We introduce an efficient method for learning linear models from uncertain data, where uncertainty is represented as a set of possible variations in the data, leading to predictive multiplicity. Our approach leverages abstract interpretation and zonotopes, a type of convex polytope, to compactly represent these dataset variations, enabling the symbolic execution of gradient descent on all possible worlds simultaneously. We develop techniques to ensure that this process converges to a fixed point and derive closed-form solutions for this fixed point. Our method provides sound over-approximations of all possible optimal models and viable prediction ranges. We demonstrate the effectiveness of our approach through theoretical and empirical analysis, highlighting its potential to reason about model and prediction uncertainty due to data quality issues in training data.
Paper Structure (75 sections, 19 theorems, 115 equations, 15 figures, 2 algorithms)

This paper contains 75 sections, 19 theorems, 115 equations, 15 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Notation, Problem Formulation and Background
  4. Learning Possible Models from Possible Worlds
  5. Sound Approximation of Possible Models with Abstract Interpretation
  6. Symbolic Abstract Domains and Zonotopes.
  7. Exact Abstract Transformers for Learning Linear Models
  8. Efficient Sound Approximation for Learning Linear Models
  9. Gradient Descent With Order Reduction
  10. Decomposition of Gradients.
  11. Experiments
  12. Datasets, Baselines, and Metrics.
  13. Robustness Verification
  14. Prediction Robustness (Uncertain Labels).
  15. Prediction Robustness (Uncertain Features).
  16. ...and 60 more sections

Key Result

Proposition 3.1

The abstract gradient descent operator $\Phi_{exact}^{\sharp}\xspace$ is an exact abstract transformer for the concrete gradient descent step operator $\Phi$. Formally, for any abstract ${{\boldsymbol{w}}^{\sharp}}\xspace$,

Figures (15)

  • Figure 1: Robustness verification on using intervals (Meyermeyer2023dataset) and zonotopes (Zorro).
  • Figure 2: Applying Zorro to causal inference. The intercept (y-axis) is the model's bias term, the treatment effect (x-axis) is the coefficient for the treatment variable.
  • Figure 3: Range of the loss, through enumeration of all possible worlds (GT) and Zorro.
  • Figure 4: Varying regularization coefficient $\lambda\xspace$: Robustness ratio (green) and worst-case test loss (red).
  • Figure 5: Robustness verification under label errors using intervals (Meyer) or zonotopes (Zorro).
  • ...and 10 more figures

Theorems & Definitions (42)

  • Definition 2.1: Possible Datasets
  • Definition 2.2: Possible Models and Prediction Range
  • Definition 2.3: Abstract Domain
  • Definition 2.4: Abstract Transformer
  • Proposition 3.1
  • Definition 3.2: Fixed Point of Abstract Gradient Descent
  • Proposition 3.3
  • Proposition 4.1
  • Theorem 4.2: Correctness of \ref{['alg:final-algo']}
  • Proposition B.1: Abstract Transformers Compose
  • ...and 32 more