Theoretical limits of descending \ell_0 sparse-regression ML algorithms

Mihailo Stojnic

arXiv: 2410.07651v1 · PDF · 10 October 2024
Categories: stat.ML, cs.IT, cs.LG, math.IT, math.ST, stat.TH
Licence: CC BY 4.0 The site shows the text.

Abstract

We study the theoretical limits of the \ell_0 (quasi) norm based optimization algorithms when employed for solving classical compressed sensing or sparse regression problems. Considering standard contexts with deterministic signals and statistical systems, we utilize Fully lifted random duality theory (Fl RDT) and develop a generic analytical program for studying performance of the maximum-likelihood (ML) decoding. The key ML performance parameter, the residual root mean square error (\textbf{RMSE}), is uncovered to exhibit the so-called phase-transition (PT) phenomenon. The associated aPT curve, which separates the regions of systems dimensions where an \ell_0 based algorithm succeeds or fails in achieving small (comparable to the noise) ML optimal \textbf{RMSE} is precisely determined as well. In parallel, we uncover the existence of another dPT curve which does the same separation but for practically feasible descending \ell_0 (d\ell_0) algorithms. Concrete implementation and practical relevance of the Fl RDT typically rely on the ability to conduct a sizeable set of the underlying numerical evaluations which reveal that for the ML decoding the Fl RDT converges astonishingly fast with corrections in the estimated quantities not exceeding \sim 0.1\% already on the third level of lifting. Analytical results are supplemented by a sizeable set of numerical experiments where we implement a simple variant of d\ell_0 and demonstrate that its practical performance very accurately matches the theoretical predictions. Completely surprisingly, a remarkably precise agreement between the simulations and the theory is observed for fairly small dimensions of the order of 100.

Items 10

Theorem

Theorem 1

Q1YD

[Stojnicflrdt23] Consider large n linear regime with \alpha\triangleq \lim_{n\rightarrow\infty} \frac{m}{n}, remaining constant as n grows. Let {\mathcal X}\subseteq {\mathbb R}^{n+1} and {\mathcal Y}\subseteq {\mathbb R}^m be two given sets and let the elements of G\in{\mathbb R}^{m\times n} be i.i.d. standard normals. Assume the complete sfl RDT frame from [Stojnicsflgscompyx23] and consider a given function f({\bf y}):R^m\rightarrow R. Set

\begin{aligned} \psi_{rp} & \triangleq - \max_{{\bf x}\in{\mathcal X}} s \max_{{\bf y}\in{\mathcal Y}} \left ( f({\bf y})+{\bf y}^TG{\bf x} \right ) \qquad \mbox{(\textcolor{blue}{\textbf{random primal}})} \\ \psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s) & \triangleq \frac{x^2y^2}{2} \sum_{k=2}^{r+1}\Bigg(\Bigg. {\bf p}_{k-1}{\bf q}_{k-1} -{\bf p}_{k}{\bf q}_{k} \Bigg.\Bigg) {\bf c}_k - \psi_{S,\infty}(f({\bf y}),{\cal X},{\cal Y},{\bf p},{\bf q},{\bf c},x,y,s) \hspace{.03in} \mbox{(\textcolor{blue}{\textbf{fl random dual}})}. \\ \end{aligned}

Let \hat{{\bf p}_0}\rightarrow 1, \hat{{\bf q}_0}\rightarrow 1, and \hat{{\bf c}_0}\rightarrow 1, \hat{{\bf p}}_{r+1}=\hat{{\bf q}}_{r+1}=\hat{{\bf c}}_{r+1}=0, and let the non-fixed parts of \hat{{\bf p}}\triangleq \hat{{\bf p}}(x,y), \hat{{\bf q}}\triangleq \hat{{\bf q}}(x,y), and \hat{{\bf c}}\triangleq \hat{{\bf c}}(x,y) be the solutions of the following system

\begin{aligned} \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf p}} = 0,\quad \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf q}} = 0,\quad \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf c}} = 0. \end{aligned}

Then,

\begin{aligned} \lim_{n\rightarrow\infty} \frac{{\mathbb E}_G \psi_{rp}}{\sqrt{n}} & = & \min_{x>0} \max_{y>0} \lim_{n\rightarrow\infty} \psi_{rd}(\hat{{\bf p}}(x,y),\hat{{\bf q}}(x,y),\hat{{\bf c}}(x,y),x,y,s) \qquad \mbox{(\textcolor{blue}{\textbf{strong sfl random duality}})}, \\ \end{aligned}

where \psi_{S,\infty}(\cdot) is as in (?)-(?).

Used by Proof, Proof

Proof of Theorem 1

Follows from the corresponding one proven in [Stojnicflrdt23] in exactly the same way as Theorem 1 in [Stojnicnegsphflrdt23].

Corollary

Corollary 1

J1II

Assume the setup of Theorem 1 with r>0, \sigma >0, c_1\in [0,\sqrt{r_1}], and {\mathcal X}={\mathcal X}(\sigma,c_1,r_1) and {\mathcal Y}={\mathbb S}^m, where {\mathcal X}(\sigma,c_1,r_1) is as in (?) and {\mathbb S}^m is the unit m-dimensional Euclidean sphere. Set r_2=\sqrt{\sigma^2+(1-2c_1+r_1^2)} and

\begin{aligned} \psi_{rp} & \triangleq - \max_{{\bf x}\in{\mathcal X}} s \max_{{\bf y}\in{\mathcal Y}} \left ( {\bf y}^TG{\bf x} \right ) \qquad \mbox{(\textcolor{blue}{\textbf{random primal}})} \\ \psi_{rd}({\bf p},{\bf q},{\bf c},r_2,1,s) & \triangleq \frac{r_2^2}{2} \sum_{k=2}^{r+1}\Bigg(\Bigg. {\bf p}_{k-1}{\bf q}_{k-1} -{\bf p}_{k}{\bf q}_{k} \Bigg.\Bigg) {\bf c}_k - \psi_{S,\infty}(0,{\cal X},{\cal Y},{\bf p},{\bf q},{\bf c},r_2,1,s) \quad \mbox{(\textcolor{blue}{\textbf{fl random dual}})}. \\ \end{aligned}

Let the non-fixed parts of \hat{{\bf p}}, \hat{{\bf q}}, and \hat{{\bf c}} be the solutions of the following system

\begin{aligned} \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},r_2,1,s)}{d{\bf p}} = 0,\quad \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},r_2,1,s)}{d{\bf q}} = 0,\quad \frac{d \psi_{rd}({\bf p},{\bf q},{\bf c},r_2,1,s)}{d{\bf c}} = 0. \end{aligned}

Then,

\begin{aligned} \lim_{n\rightarrow\infty} \frac{{\mathbb E}_G \psi_{rp}}{\sqrt{n}} & = & \lim_{n\rightarrow\infty} \psi_{rd}(\hat{{\bf p}},\hat{{\bf q}},\hat{{\bf c}},r_2,1,s) \qquad \mbox{(\textcolor{blue}{\textbf{strong sfl random duality}})}, \\ \end{aligned}

where \psi_{S,\infty}(\cdot) is as in (?)-(?).

Used by Proof

Proof of Corollary 1

Follows immediately as a direct consequence of Theorem 1, after choosing f({\bf y})=0 and recognizing that the specialized sets {\mathcal X}={\mathcal X}(\sigma,c_1,r_1) and {\mathcal Y}={\mathbb S}^m are such that their elements have r_2=\sqrt{\sigma^2+(1-2c_1+r_1^2)} and unit Euclidean norms, respectively.

Uses Theorem 1

Theorem

Theorem 2

0XA0

Assume the setup of Theorem 1 and Corollary 1, and consider large n linear regime with \alpha=\lim_{n\rightarrow\infty} \frac{m}{n}. Set

\begin{aligned} \varphi(D,{\bf c}) & = & {\mathbb E}_{G,{\mathcal U}_{r+1}} \frac{1}{{\bf c}_r} \log \left ( {\mathbb E}_{{\mathcal U}_{r}} \left ( \dots \left ( {\mathbb E}_{{\mathcal U}_3}\left (\left ({\mathbb E}_{{\mathcal U}_2} \left ( \left ( e^{D} \right )^{{\bf c}_2}\right )\right )^{\frac{{\bf c}_3}{{\bf c}_2}}\right )\right )^{\frac{{\bf c}_4}{{\bf c}_3}} \dots \right )^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right ), \\ \end{aligned}

and

\begin{aligned} \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1) & \triangleq & \frac{r_2^2}{2} \sum_{k=2}^{r+1}\Bigg(\Bigg. {\bf p}_{k-1}{\bf q}_{k-1} -{\bf p}_{k}{\bf q}_{k} \Bigg.\Bigg) {\bf c}_k - \frac{\sigma^2}{2} \sum_{k=2}^{r+1}\Bigg(\Bigg. {\bf q}_{k-1} -{\bf q}_{k} \Bigg.\Bigg) {\bf c}_k \\ & & - \nu_1 c_1 -\gamma r_1^2 -\nu \beta - (1-\beta)\varphi(D_{10}^{(per)}(-1),{\bf c}) - \frac{1}{n}\sum_{i=1}^{k}\varphi(D_{i1}^{(per)}(-1),{\bf c}) \\ & & + \gamma_{sq}r_2 - \alpha\varphi(D_1^{(sph)}(-1),{\bf c}), \end{aligned}

where D_{i1}^{(per)}(-1), D_{10}^{(per)}(-1), and D_1^{(sph)}(-1) are as in (?), (?), and (?), respectively. Let the “fixed” parts of \hat{{\bf p}}, \hat{{\bf q}}, and \hat{{\bf c}} satisfy \hat{{\bf p}}_1\rightarrow 1, \hat{{\bf q}}_1\rightarrow 1, \hat{{\bf c}}_1\rightarrow 1, \hat{{\bf p}}_{r+1}=\hat{{\bf q}}_{r+1}=\hat{{\bf c}}_{r+1}=0. Further, let the “non-fixed” parts of \hat{{\bf p}}_k, \hat{{\bf q}}_k, and \hat{{\bf c}}_k (k\in\{2,3,\dots,r\}) and \hat{\gamma}_{sq}, \hat{\gamma}, \hat{\nu}_1, and \hat{\nu} be the solutions of the following system of equations

\begin{aligned} \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d{\bf p}} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d{\bf q}} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d{\bf c}} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d\gamma_{sq}} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d\gamma} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d\nu_1} & = & 0 \\ \frac{d \bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},\gamma,\nu_1,\nu,\sigma,c_1,r_1)}{d\nu} & = & 0. \end{aligned}

Consequently, let

\begin{aligned} c_k(\hat{{\bf p}},\hat{{\bf q}}) & = & \sqrt{\hat{{\bf q}}_{k-1}-\hat{{\bf q}}_k} \\ b_k(\hat{{\bf p}},\hat{{\bf q}}) & = & \sqrt{\hat{{\bf p}}_{k-1}-\hat{{\bf p}}_k}. \end{aligned}

Then

\sigma\xi(c_1.r_1) = \bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},\hat{{\bf c}},\hat{\gamma}_{sq},\hat{\gamma},\hat{\nu_1},\hat{\nu},\sigma,c_1,r_1).

Proof of Theorem 2

Follows from Theorem 1, Corollary 1, the sfl RDT machinery presented in [Stojnicnflgscompyx23, Stojnicsflgscompyx23, Stojnicflrdt23, Stojnichopflrdt23], the above discussion, and after integrating out \left (\sum_{k=2}^{r+1}c_k{\bf h}_{n+1}^{(k)}\right )\sigma\sqrt{n}.

Uses Theorem 1, Corollary 1

Corollary

Corollary 2

JPA1

Assume the setup of Theorem 2 (and implicitly Theorem1 and Corollary 1). Set r=2. Then

\begin{aligned} \hat{\gamma}_{sq} & = & \frac{1}{2}\frac{1-\hat{{\bf p}}_2}{1-\hat{{\bf q}}_2}\sqrt{\frac{\hat{{\bf q}}_2}{\hat{{\bf p}}_2}}\sqrt{\alpha} \\ \hat{{\bf c}}_2 & = & \frac{1}{r_2} \left ( \frac{1}{1-\hat{{\bf p}}_2} \sqrt{\frac{\hat{{\bf p}}_2}{\hat{{\bf q}}_2}} - \frac{1}{1-\hat{{\bf q}}_2} \sqrt{\frac{\hat{{\bf q}}_2}{\hat{{\bf p}}_2}}\right ) \sqrt{\alpha}. \end{aligned}

Proof of Corollary 2

Both equalities follow as analogues to (87) and (90) in [Stojnicnegsphflrdt23] after line by line repetitions of their derivations with minimal cosmetic sign and r_2 adjustments.

Corollary

Corollary 3

23WL

Assume the setup of Theorem 2 (and implicitly Theorem1 and Corollary 1). Set r=2. Then

\begin{aligned} \hat{\gamma}_{sq} & = & \frac{1}{2}\frac{1-\hat{{\bf p}}_2}{1-\hat{{\bf q}}_2}\frac{\hat{{\bf q}}_2-\hat{{\bf q}}_3}{\hat{{\bf p}}_2-\hat{{\bf p}}_3}\sqrt{\frac{\hat{{\bf p}}_3}{\hat{{\bf q}}_3}}\sqrt{\alpha} \\ \hat{{\bf c}}_2 & = & \frac{1}{r_2} \left ( \frac{1}{1-\hat{{\bf p}}_2} \frac{\hat{{\bf p}}_2-\hat{{\bf p}}_3}{\hat{{\bf q}}_2-\hat{{\bf q}}_3} \sqrt{\frac{\hat{{\bf q}}_3}{\hat{{\bf p}}_3}} - \frac{1}{1-\hat{{\bf q}}_2} \frac{\hat{{\bf q}}_2-\hat{{\bf q}}_3}{\hat{{\bf p}}_2-\hat{{\bf p}}_3} \sqrt{\frac{\hat{{\bf p}}_3}{\hat{{\bf q}}_3}} \right ) \sqrt{\alpha} \\ \hat{{\bf c}}_3 & = & \frac{1}{r_2} \left ( \frac{1}{\hat{{\bf p}}_2-\hat{{\bf p}}_3} \sqrt{\frac{\hat{{\bf p}}_3}{\hat{{\bf q}}_3}} - \frac{1}{\hat{{\bf q}}_2-\hat{{\bf q}}_3} \sqrt{\frac{\hat{{\bf q}}_3}{\hat{{\bf p}}_3}} \right ) \sqrt{\alpha}. \end{aligned}

Proof of Corollary 3

All three equalities follow as analogues to (142), (145), and (147) in [Stojnicnegsphflrdt23] after line by line repetitions of their derivations.

Source and licence

The text on this page is from arXiv:2410.07651v1 by Mihailo Stojnic, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2410/2410.07651.jsonl.