Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inference From Random Restarts

Moeen Nehzati, Diego Cussen

TL;DR

This work provides a simple probabilistic framework for interpreting repeated convergence to the same output as evidence of a unique solution or a dominant basin of attraction, and adopts a Bayesian approach to infer basin sizes and the probability of solution uniqueness from repeated identical outputs.

Abstract

Algorithms for computing equilibria, optima, and fixed points in nonconvex problems often depend sensitively on practitioner-chosen initial conditions. When uniqueness of a solution is of interest, a common heuristic is to run such algorithms from many randomly selected initial conditions and to interpret repeated convergence to the same output as evidence of a unique solution or a dominant basin of attraction. Despite its widespread use, this practice lacks a formal inferential foundation. We provide a simple probabilistic framework for interpreting such numerical evidence. First, we give sufficient conditions under which an algorithm's terminal output is a measurable function of its initial condition, allowing probabilistic reasoning over outcomes. Second, we provide sufficient conditions ensuring that an algorithm admits only finitely many possible terminal outcomes. While these conditions may be difficult to verify on a case-by-case basis, we give simple sufficient conditions for broad classes of problems under which almost all instances admit only finitely many outcomes (in the sense of prevalence). Standard algorithms such as gradient descent and damped fixed-point iteration applied to sufficiently smooth functions satisfy these conditions. Within this framework, repeated solver runs correspond to independent samples from the induced distribution over outcomes. We adopt a Bayesian approach to infer basin sizes and the probability of solution uniqueness from repeated identical outputs, and we establish convergence rates for the resulting posterior beliefs. Finally, we apply our framework to settings in the existing industrial organization literature, where random-restart heuristics are used. Our results formalize and qualify these arguments, clarifying when repeated convergence provides meaningful evidence for uniqueness and when it does not.

Inference From Random Restarts

TL;DR

This work provides a simple probabilistic framework for interpreting repeated convergence to the same output as evidence of a unique solution or a dominant basin of attraction, and adopts a Bayesian approach to infer basin sizes and the probability of solution uniqueness from repeated identical outputs.

Abstract

Algorithms for computing equilibria, optima, and fixed points in nonconvex problems often depend sensitively on practitioner-chosen initial conditions. When uniqueness of a solution is of interest, a common heuristic is to run such algorithms from many randomly selected initial conditions and to interpret repeated convergence to the same output as evidence of a unique solution or a dominant basin of attraction. Despite its widespread use, this practice lacks a formal inferential foundation. We provide a simple probabilistic framework for interpreting such numerical evidence. First, we give sufficient conditions under which an algorithm's terminal output is a measurable function of its initial condition, allowing probabilistic reasoning over outcomes. Second, we provide sufficient conditions ensuring that an algorithm admits only finitely many possible terminal outcomes. While these conditions may be difficult to verify on a case-by-case basis, we give simple sufficient conditions for broad classes of problems under which almost all instances admit only finitely many outcomes (in the sense of prevalence). Standard algorithms such as gradient descent and damped fixed-point iteration applied to sufficiently smooth functions satisfy these conditions. Within this framework, repeated solver runs correspond to independent samples from the induced distribution over outcomes. We adopt a Bayesian approach to infer basin sizes and the probability of solution uniqueness from repeated identical outputs, and we establish convergence rates for the resulting posterior beliefs. Finally, we apply our framework to settings in the existing industrial organization literature, where random-restart heuristics are used. Our results formalize and qualify these arguments, clarifying when repeated convergence provides meaningful evidence for uniqueness and when it does not.
Paper Structure (21 sections, 32 theorems, 119 equations, 3 tables)

This paper contains 21 sections, 32 theorems, 119 equations, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bayesian Inference For Discrete Latent Distributions
  4. Basins of Attraction and Algorithmic Geometry
  5. The Gap Between Geometry and Inference
  6. Model
  7. Measurable Dynamic Solvers
  8. Finitely Many Reachable Terminal States
  9. Prevalence of Finitely Many Reachable Terminal States
  10. Numerical inaccuracies
  11. Inference
  12. Basins of Attraction
  13. Number of Reachable Terminal States
  14. Spike and Slab
  15. Mixture of Finite Models
  16. ...and 6 more sections

Key Result

Lemma 3.3

If $X$ is closed and convex (Assumption ass:convex-closed) and $Q$ is Lipschitz continuous, ass:well-behaved-proj is satisfied.

Theorems & Definitions (81)

  • Definition 3.1: Tangent and Normal Cones
  • Definition 3.2: Projected Autonomous Dynamics
  • Remark 3.1
  • Remark 3.2: $Q^{\Pi_{\operatorname{TC}_X}}(x)=Q(x)$ on $\operatorname{int} X$
  • Lemma 3.3
  • proof
  • Definition 3.4: Dynamic solver
  • Definition 3.5: Reachable terminal states (RTS) and $\mu$-observable RTS
  • Proposition 3.5
  • proof : Proof sketch
  • ...and 71 more