Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generic linear convergence for algorithms of non-linear least squares over smooth varieties

Shenglong Hu, Ke Ye

TL;DR

It is proved that under some mild assumptions, these troublesome singularities of the image can always be avoided, which enables us to establish a linear convergence rate for iterative sequences generated by algorithms satisfying some standard assumptions.

Abstract

In applications, a substantial number of problems can be formulated as non-linear least squares problems over smooth varieties. Unlike the usual least squares problem over a Euclidean space, the non-linear least squares problem over a variety can be challenging to solve and analyze, even if the variety itself is simple. Geometrically, this problem is equivalent to projecting a point in the ambient Euclidean space onto the image of the given variety under a non-linear map. It is the singularities of the image that make both the computation and the analysis difficult. In this paper, we prove that under some mild assumptions, these troublesome singularities can always be avoided. This enables us to establish a linear convergence rate for iterative sequences generated by algorithms satisfying some standard assumptions. We apply our general results to the low-rank partially orthogonal tensor approximation problem. As a consequence, we obtain the linear convergence rate for a classical APD-ALS method applied to a generic tensor, without any further assumptions.

Generic linear convergence for algorithms of non-linear least squares over smooth varieties

TL;DR

It is proved that under some mild assumptions, these troublesome singularities of the image can always be avoided, which enables us to establish a linear convergence rate for iterative sequences generated by algorithms satisfying some standard assumptions.

Abstract

In applications, a substantial number of problems can be formulated as non-linear least squares problems over smooth varieties. Unlike the usual least squares problem over a Euclidean space, the non-linear least squares problem over a variety can be challenging to solve and analyze, even if the variety itself is simple. Geometrically, this problem is equivalent to projecting a point in the ambient Euclidean space onto the image of the given variety under a non-linear map. It is the singularities of the image that make both the computation and the analysis difficult. In this paper, we prove that under some mild assumptions, these troublesome singularities can always be avoided. This enables us to establish a linear convergence rate for iterative sequences generated by algorithms satisfying some standard assumptions. We apply our general results to the low-rank partially orthogonal tensor approximation problem. As a consequence, we obtain the linear convergence rate for a classical APD-ALS method applied to a generic tensor, without any further assumptions.

Paper Structure

This paper contains 19 sections, 27 theorems, 79 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notations and preliminaries
  3. Matrix manifolds
  4. Morse function
  5. Two inequalities
  6. Partially orthogonal tensor
  7. Polar decomposition
  8. Locations of critical points for generic data
  9. Examples
  10. Projection onto a hyperboloid
  11. LU approximation of $2\times 2$ matrix
  12. Low rank tensor approximation
  13. Case $r = 1$
  14. Case $r = 2$
  15. Low rank partially orthogonal tensor approximation
  16. ...and 4 more sections

Key Result

Lemma 2.1

M-63 Let $\mathrm{M}$ be a submanifold of $\mathbb R^n$. For a generic $\mathbf a = (a_1,\dots,a_n)^{\mathsf{T}} \in\mathbb R^n$, the squared Euclidean distance function $f(\mathbf x) \coloneqq \|\mathbf x-\mathbf a\|^2$ is a Morse function on $\mathrm{M}$.

Figures (1)

  • Figure 1: Incidence variety and its projections

Theorems & Definitions (40)

  • Lemma 2.1: Projection is Generically Morse
  • Proposition 2.2: Ł ojasiewicz's Gradient Inequality
  • Lemma 2.3: Hoffman's Error Bound
  • Proposition 2.4: Polar Decomposition
  • Lemma 3.1
  • proof
  • Lemma 3.2: Incidence variety
  • proof
  • Lemma 3.3
  • Lemma 3.4
  • ...and 30 more