A note on time-uniform concentration inequality for matrix products
Tuan Pham, Alessandro Rinaldo
TL;DR
The paper addresses time-uniform concentration bounds for the matrix product $Z_n=\prod_{i=n}^{1}(\bm{I}_d+\eta_i X_i)$ with i.i.d. PSD $X_i$ and $\mathbb{E}X_i=\Sigma$. It employs a submartingale argument, introducing $\bm{Y}_n=\bm{E}_n^{-1}\bm{Z}_n-\bm{I}_d$ and a geometric-epoch decomposition to enable Doob-type maximal inequalities, providing a simple any-time bound in this matrix-product setting. Under the condition $M_n\sqrt{2V_n\log(d/\delta)}\le1$ and for any $\eta>1$ and function $h$ with $\sum 1/h(k)\le1$, it proves $\mathbb{P}(\exists n:\|\bm{Z}_n-\bm{E}_n\|\ge t(\delta/h(k_n),\lfloor\eta^{k_n+1}\rfloor))\le \delta$, with $t(\delta,n)=e\|\bm{E}_n\|M_n\sqrt{2V_n\log(d/\delta)}$. The boundary can be smoothed and is accompanied by an extra factor $\|\bm{E}_n\|$ (potentially replaceable by $M_n$); a bounded-data example shows alignment with fixed-time results. The work is motivated by the analysis of Oja's algorithms.
Abstract
This short note contains a simple argument that allows us to go from fixed-time to any-time bounds for the concentration of matrix products. The result presented here is motivated by the analysis of Oja's algorithms.