Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Piecewise Polynomial Regression of Tame Functions via Integer Programming

Gilles Bareilles, Johannes Aspman, Jiri Nemecek, Jakub Marecek

TL;DR

This work tackles accurate approximation of tame functions, definable in o-minimal structures, by piecewise polynomial functions over a domain $A$. It establishes a rigorous approximation bound that combines smooth-polynomial approximation on full-dimensional cells with a Lipschitz-based treatment of cell interfaces, yielding $||f-p||_{\infty,A} \le C_1 N^{-m} + C_2 l^{-2/(n-1)}$ for a piecewise polynomial family $\widetilde{P}_{N}^{l}(A)$. A novel mixed-integer programming formulation extends the OCT-H framework to regression with arbitrary-degree polynomials and affine-hyperplane partitions, enabling global optimization of the piecewise polynomial fit from sampled data. The authors demonstrate practical viability on tame neural networks, cone-like functions, and denoising tasks, while acknowledging scalability limitations and outlining avenues for improving optimization efficiency and extending applicability to higher dimensions. Overall, the paper provides both theoretical guarantees and a concrete optimization-based pipeline for estimating tame functions via hierarchical, piecewise-polynomial representations with potential impact across machine learning and optimization domains.

Abstract

Tame functions are a class of nonsmooth, nonconvex functions, which feature in a wide range of applications: functions encountered in the training of deep neural networks with all common activations, value functions of mixed-integer programs, or wave functions of small molecules. We consider approximating tame functions with piecewise polynomial functions. We bound the quality of approximation of a tame function by a piecewise polynomial function with a given number of segments on any full-dimensional cube. We also present the first mixed-integer programming formulation of piecewise polynomial regression. Together, these can be used to estimate tame functions. We demonstrate promising computational results.

Piecewise Polynomial Regression of Tame Functions via Integer Programming

TL;DR

This work tackles accurate approximation of tame functions, definable in o-minimal structures, by piecewise polynomial functions over a domain . It establishes a rigorous approximation bound that combines smooth-polynomial approximation on full-dimensional cells with a Lipschitz-based treatment of cell interfaces, yielding for a piecewise polynomial family . A novel mixed-integer programming formulation extends the OCT-H framework to regression with arbitrary-degree polynomials and affine-hyperplane partitions, enabling global optimization of the piecewise polynomial fit from sampled data. The authors demonstrate practical viability on tame neural networks, cone-like functions, and denoising tasks, while acknowledging scalability limitations and outlining avenues for improving optimization efficiency and extending applicability to higher dimensions. Overall, the paper provides both theoretical guarantees and a concrete optimization-based pipeline for estimating tame functions via hierarchical, piecewise-polynomial representations with potential impact across machine learning and optimization domains.

Abstract

Tame functions are a class of nonsmooth, nonconvex functions, which feature in a wide range of applications: functions encountered in the training of deep neural networks with all common activations, value functions of mixed-integer programs, or wave functions of small molecules. We consider approximating tame functions with piecewise polynomial functions. We bound the quality of approximation of a tame function by a piecewise polynomial function with a given number of segments on any full-dimensional cube. We also present the first mixed-integer programming formulation of piecewise polynomial regression. Together, these can be used to estimate tame functions. We demonstrate promising computational results.
Paper Structure (29 sections, 7 theorems, 39 equations, 10 figures, 7 tables)

This paper contains 29 sections, 7 theorems, 39 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Our contributions.
  3. Background on tame geometry
  4. Definability in o-minimal structures
  5. Tame functions are piecewise Cr
  6. Approximation of tame functions
  7. Proof outline of \ref{['th:main']}.
  8. Mixed-integer formulation of piecewise polynomial approximation
  9. Numerical experiments
  10. Setup.
  11. Regression of tame functions
  12. Piecewise-affine function.
  13. Non-piecewise-linear Neural Network
  14. Denoising of a piecewise-constant 2d signal
  15. Notes on the optimization process
  16. ...and 14 more sections

Key Result

Proposition 1

Fix an o-minimal expansion of $\mathbb{R}$. Consider a definable full-dimensional set $A\subset\mathbb{R}^{n}$ and a definable function $f:A \to \mathbb{R}$. Then, for any positive integer $m$, there exists a finite collection $\mathcal{W}$ of sets $\mathcal{M}\subset\mathbb{R}^{n}$, called cells, s

Figures (10)

  • Figure 1: Left pane: a generic 2-dimensional function, with level lines (white) and nonsmooth points (black). Right pane: piecewise polynomial approximation of the network, obtained from the proposed integer program with a depth 3 regression tree and degree 2 polynomials.
  • Figure 2: Illustration of the "cone" function \ref{['eq:cone2dintro']}, with $s^\text{cone}=r^\text{cone}=0.5$, showing (i) the level lines of the function, and (ii) the decomposition of the domain into cells on which the function is smooth, as provided by \ref{['prop:celldecomp']}; see \ref{['table:conecells']} for details.
  • Figure 3: Binary tree and corresponding partition.
  • Figure 4: Tame functions and regression results for trees of depth 2 and 3. Red crosses are the coordinates of samples used for the training. Black lines show the decomposition of the space.
  • Figure 5: Four denoising scenarios from NEURIPS2021_dba4c1a1. The regression trees are restricted to axis-aligned splits and have depth 4. First row: ground truth, second row: (corrupted) regression signal, third row: recovered signal from the mixed integer formulation.
  • ...and 5 more figures

Theorems & Definitions (22)

  • Example 1: Analytic-exponential structure
  • Proposition 1: $\mathcal{C}^{m}$-cell decomposition
  • proof
  • Example 2
  • Definition 1: Polynomial functions
  • Definition 2: Piecewise-polynomial functions
  • Theorem 1: Main result
  • Remark 1: Axis-aligned regression
  • Definition 3: o-minimal structure
  • Example 3: Semialgebraic structure
  • ...and 12 more