Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sharp Convergence Rates for Matching Pursuit

Jason M. Klusowski, Jonathan W. Siegel

TL;DR

This work resolves a long-standing question about the convergence rate of the pure greedy matching pursuit when approximating a function in the variation space $\mathcal{K}_1(\mathbb{D})$. By constructing a carefully designed worst-case dictionary on $H=\ell^2$ and a residual sequence with norm $\|r_n\|=(n+1)^{-1/2+\beta}$, the authors establish a sharp lower bound showing the decay $\|f-f_n\| \ge C\|f\|_{\mathcal{K}_1(\mathbb{D})}n^{-\alpha}$ with $\alpha=\gamma/(2(2+\gamma))\approx 0.182$, where $\gamma>1$ solves a nonlinear equation. They further demonstrate that any shrinkage $0<s<1$ improves the worst-case rate, and prove that the known upper bounds are tight up to this exponent. The analysis hinges on a detailed constructive approach (via a parameterized dictionary and a smoothed function $\phi$) and a suite of asymptotic and integral-inequality arguments, culminating in a Schauder-fixed-point proof to realize the required function $\phi$ and the associated exponents. The results advance the understanding of non-linear dictionary approximation and delineate the limits of pure greedy strategies relative to other greedy variants.

Abstract

We study the fundamental limits of matching pursuit, or the pure greedy algorithm, for approximating a target function $ f $ by a linear combination $f_n$ of $n$ elements from a dictionary. When the target function is contained in the variation space corresponding to the dictionary, many impressive works over the past few decades have obtained upper and lower bounds on the error $\|f-f_n\|$ of matching pursuit, but they do not match. The main contribution of this paper is to close this gap and obtain a sharp characterization of the decay rate, $n^{-α}$, of matching pursuit. Specifically, we construct a worst case dictionary which shows that the existing best upper bound cannot be significantly improved. It turns out that, unlike other greedy algorithm variants which converge at the optimal rate $ n^{-1/2}$, the convergence rate $n^{-α}$ is suboptimal. Here, $α\approx 0.182$ is determined by the solution to a certain non-linear equation.

Sharp Convergence Rates for Matching Pursuit

TL;DR

This work resolves a long-standing question about the convergence rate of the pure greedy matching pursuit when approximating a function in the variation space . By constructing a carefully designed worst-case dictionary on and a residual sequence with norm , the authors establish a sharp lower bound showing the decay with , where solves a nonlinear equation. They further demonstrate that any shrinkage improves the worst-case rate, and prove that the known upper bounds are tight up to this exponent. The analysis hinges on a detailed constructive approach (via a parameterized dictionary and a smoothed function ) and a suite of asymptotic and integral-inequality arguments, culminating in a Schauder-fixed-point proof to realize the required function and the associated exponents. The results advance the understanding of non-linear dictionary approximation and delineate the limits of pure greedy strategies relative to other greedy variants.

Abstract

We study the fundamental limits of matching pursuit, or the pure greedy algorithm, for approximating a target function by a linear combination of elements from a dictionary. When the target function is contained in the variation space corresponding to the dictionary, many impressive works over the past few decades have obtained upper and lower bounds on the error of matching pursuit, but they do not match. The main contribution of this paper is to close this gap and obtain a sharp characterization of the decay rate, , of matching pursuit. Specifically, we construct a worst case dictionary which shows that the existing best upper bound cannot be significantly improved. It turns out that, unlike other greedy algorithm variants which converge at the optimal rate , the convergence rate is suboptimal. Here, is determined by the solution to a certain non-linear equation.
Paper Structure (10 sections, 11 theorems, 184 equations, 1 figure)

This paper contains 10 sections, 11 theorems, 184 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Construction of the Worst-Case Dictionary
  3. Basic Construction
  4. Proof of Proposition \ref{['first-part-main-proposition']}
  5. Inner Product Formulas
  6. Parameter Asymptotics
  7. Estimates for the Conditions \ref{['conditions-1']}
  8. Estimates for the Inner Product Conditions
  9. Proof of Proposition \ref{['second-part-main-proposition']}
  10. Acknowledgements

Key Result

Theorem 1

Let $\gamma > 1$ be the root of the equation gamma-equation. Then for every $\alpha > \gamma/(2(2+\gamma))$, there exists a dictionary $\mathbb{D}\subset H$ and a function $f\in \mathcal{K}_1(\mathbb{D})$ such that the iterates of the pure greedy algorithm (i.e., matching pursuit) pure-greedy-algori

Figures (1)

  • Figure 1: Left: The first $3$ iterates of $\tilde{T}_G$, which demonstrate that $f_3 > 0$. Right: The numerically calculated solution to \ref{['differential-integral-equation']} for the $G$ corresponding to $\beta^*$ and $\tau^*$ for $s=1$.

Theorems & Definitions (19)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Lemma 1
  • proof
  • Proposition 3
  • proof
  • Lemma 2
  • proof
  • Proposition 4
  • ...and 9 more