arXiv reader

Joint-sparse recovery from multiple measurements

Ewout van den Berg and Michael P. Friedlander

arXiv
0904.2051v1 · PDF · 14 April 2009
Categories
cs.IT, math.IT
Journal
IEEE Transactions on Information Theory, 56(5):2516-2527, 2010
DOI
10.1109/TIT.2010.2043876
Licence
CC BY 4.0 The site shows the text.

Abstract

The joint-sparse recovery problem aims to recover, from sets of compressed measurements, unknown sparse matrices with nonzero entries restricted to a subset of rows. This is an extension of the single-measurement-vector (SMV) problem widely studied in compressed sensing. We analyze the recovery properties for two types of recovery algorithms. First, we show that recovery using sum-of-norm minimization cannot exceed the uniform recovery rate of sequential SMV using \ell_1 minimization, and that there are problems that can be solved with one approach but not with the other. Second, we analyze the performance of the ReMBo algorithm [M. Mishali and Y. Eldar, IEEE Trans. Sig. Proc., 56 (2008)] in combination with \ell_1 minimization, and show how recovery improves as more measurements are taken. From this analysis it follows that having more measurements than number of nonzero rows does not improve the potential theoretical recovery rate.

Items 13

Notation

Notation

R599

We assume throughout that A is a full-rank matrix in \mathbb{R}^{m\times n}, and that X_0 is an s row-sparse matrix in \mathbb{R}^{n\times r}. We follow the convention that all vectors are column vectors. For an arbitrary matrix M, its jth column is denoted by the column vector M\ensuremath{^{\scalebox{.6}{}{j}}}; its ith row is the transpose of the column vector M\ensuremath{^{{i}{\scalebox{.6}{}}}}. The ith entry of a vector v is denoted by v_i. We make exceptions for e_i = I\ensuremath{^{\scalebox{.6}{}{i}}} and for x_0 (resp., X_0), which represents the sparse vector (resp., matrix) we want to recover. When there is no ambiguity we sometimes write m_i to denote M\ensuremath{^{\scalebox{.6}{}{i}}}. When concatenating vectors into matrices, [a,b,c] denotes horizontal concatenation and [a;b;c] denotes vertical concatenation. When indexing with \mathcal{I}, we define the vector v_{\mathcal{I}} := [v_i]_{i\in\mathcal{I}}, and the m\times\ensuremath{|\mathcal{I}|} matrix A_{\mathcal{I}} := [A\ensuremath{^{\scalebox{.6}{}{j}}}]_{j\in\mathcal{I}}. Row or column selection takes precedence over all other operators.

Theorem

Theorem 0 (Donoho and Elad , Gribonval and Nielsen)

UM9Q

Let A be an m\times n matrix and \mathcal{I} \subseteq \{1,\ldots,n\} be a fixed index set. Then all x_0\in\mathbb{R}^n with \ensuremath{\textrm{Supp}}(x_0) \subseteq \mathcal{I} can be uniquely recovered from b = Ax_0 using basis pursuit (?) if and only if for all z \in \ensuremath{\textrm{Ker}}(A)\setminus \{0\},

\sum_{j\in\mathcal{I}} |z_j| < \sum_{j\not\in\mathcal{I}} |z_j|.

That is, the \ell_1-norm of z on \mathcal{I} is strictly less than the \ell_1-norm of z on the complement \mathcal{I}^c.

Used by Proof
Theorem

Theorem 0

0U14

Let A be an m\times n matrix, k be a positive integer, \mathcal{I} \subseteq \{1,\ldots,n\} be a fixed index set, and let \|\cdot\| denote any vector norm. Then all X_0 \in \mathbb{R}^{n\times r} with \ensuremath{\textrm{Supp}}_{\mathrm{row}}(X_0) \subseteq\mathcal{I} can be uniquely recovered from B = AX_0 using (?) if and only if for all Z with columns Z\ensuremath{^{\scalebox{.6}{}{k}}} \in \ensuremath{\textrm{Ker}}(A)\setminus\{0\},

\sum_{j\in\mathcal{I}}\|Z\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\| < \sum_{j\not\in\mathcal{I}}\|Z\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\|.
Used by Proof
Proof of Theorem 0

For the “only if” part, suppose that there is a Z with columns Z\ensuremath{^{\scalebox{.6}{}{k}}}\in\ensuremath{\textrm{Ker}}(A)\setminus\{0\} such that (0) does not hold. Now, choose X\ensuremath{^{{j}{\scalebox{.6}{}}}} = Z\ensuremath{^{{j}{\scalebox{.6}{}}}} for all j\in\mathcal{I} and with all remaining rows zero. Set B = AX. Next, define V = X - Z, and note that AV = AX - AZ = AX = B. The construction of V implies that \sum_{j}\|X\ensuremath{^{{j}{\scalebox{.6}{}}}}\| \geq \sum_{j}\|V\ensuremath{^{{j}{\scalebox{.6}{}}}}\|, and consequently X cannot be the unique solution of (?).

Conversely, let X be an arbitrary matrix with \ensuremath{\textrm{Supp}}_{\mathrm{row}}(X)\subseteq \mathcal{I}, and let B = AX. To show that X is the unique solution of (?) it suffices to show that for any Z with columns Z\ensuremath{^{\scalebox{.6}{}{k}}}\in\ensuremath{\textrm{Ker}}(A)\setminus\{0\},

\sum_{j}\|(X+Z)\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\| > \sum_{j}\|X\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\|.

This is equivalent to

\sum_{j\not\in\mathcal{I}}\|Z\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\| + \sum_{j\in\mathcal{I}}\|(X+Z)\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\| - \sum_{j\in\mathcal{I}}\|X\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\| > 0.

Applying the reverse triangle inequality, \|a+b\| - \|b\| \geq -\|a\|, to the summation over j\in\mathcal{I} and reordering exactly gives condition (0).

Theorem

Theorem 0

DTZJ

Let A be an m\times n matrix, r be a positive integer, \mathcal{I} \subseteq \{1,\ldots,n\} be a fixed index set, and \|\cdot\| denote any vector norm. Then uniform recovery of all X \in \mathbb{R}^{n\times r} with \ensuremath{\textrm{Supp}}_{\mathrm{row}}(X) \subseteq \mathcal{I} using sums of norms (?) implies uniform recovery on \mathcal{I} using \ell_{1,1}.

Proof of Theorem 0

For uniform recovery on support \mathcal{I} to hold it follows from Theorem 0 that for any matrix Z with columns Z\ensuremath{^{\scalebox{.6}{}{k}}}\in\ensuremath{\textrm{Ker}}(A)\setminus\{0\}, property (0) holds. In particular it holds for Z with Z\ensuremath{^{\scalebox{.6}{}{k}}} = \skew{2.8}\bar z for all k, with \skew{2.8}\bar z\in\ensuremath{\textrm{Ker}}(A)\setminus\{0\}. Note that for these matrices there exist a norm-dependent constant \gamma such that

|\skew{2.8}\bar z_j| = \gamma \|Z\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\|.

Since the choice of \skew{2.8}\bar z was arbitrary, it follows from (0) that the NS-condition (0) for independent recovery of vectors B\ensuremath{^{\scalebox{.6}{}{k}}} using \ell_1 in Theorem 0 is satisfied. Moreover, because \ell_{1,1} is equivalent to independent recovery, we also have uniform recovery on \mathcal{I} using \ell_{1,1}.

Theorem

Theorem 0

IW5S

Let A be an m\times n matrix, and B be an m\times r matrix. Then a set of sufficient conditions for X to be the unique minimizer of (?) with Lagrange multiplier Y\in\mathbb{R}^{m\times r} and row support \mathcal{I}=\ensuremath{\textrm{Supp}}_{\mathrm{row}}(X), is that

\begin{alignat}{2} & AX=B, \\ & (A^T\! Y)\ensuremath{^{\scalebox{.6}{$\downarrow$}{j}}} = (X^*)\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}/\|(X^*)\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}}\|_2, &\qquad j &\in\mathcal{I} \\ &\|(A^T\! Y)\ensuremath{^{\scalebox{.6}{$\downarrow$}{j}}}\|_2 < 1, &\qquad j &\not\in\mathcal{I} \\ &\mathop{\hbox{\rm rank}}(A_\mathcal{I}) = \vert\mathcal{I}\vert. \end{alignat}
Proof of Theorem 0

The first three conditions clearly imply that (X,Y) primal and dual feasible, and thus satisfy (?) and (?). Conditions (0) and (0) together imply that

\mathop{\hbox{\rm trace}}(B^T\! Y) \equiv \sum_{j=1}^n[(A^T\! Y)\ensuremath{^{\scalebox{.6}{$\downarrow$}{j}}}]^T X\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}} = \sum_{j=1}^n X\ensuremath{^{{j}{\scalebox{.6}{$\rightarrow$}}}} \equiv \|X\|_{1,2}.

The first and last identities above follow directly from the definitions of the matrix trace and of the norm \|\cdot\|_{1,2}, respectively; the middle equality follows from the standard Cauchy inequality. Thus, the zero-gap requirement (?) is satisfied. The conditions (0)–(0) are therefore sufficient for (X,Y) to be an optimal primal-dual solution of (?). Because Y determines the support and is a Lagrange multiplier for every solution X, this support must be unique. It then follows from condition (0) that X must be unique.

Corollary

Corollary 0

VK00

Recovery using \ell_{1,2} is generally not only characterized by the row-support and the sign pattern of the nonzero entries in X_0, but also by the magnitude of the nonzero entries.

Theorem

Theorem 0

77BB

Let C(n,d) denote the maximum attainable number of orthant interiors intersected by a hyperplane in \mathbb{R}^n generated by d vectors. Then C(n,1) = 2, C(n,d) = 2^n for d \geq n. In general, C(n,d) is given by

C(n,d) = C(n-1,d-1) + C(n-1,d) = 2 \sum_{i=0}^{d-1}{n-1 \choose i}.
Proof of Theorem 0

The number of intersected orthants is exactly equal to the number of proper sign patterns (excluding zero values) that can be generated by linear combinations of those d vectors. When d=1, there can only be two such sign patterns corresponding to positive and negative multiples of that vector, thus giving C(n,1) = 2. Whenever d \geq n, we can choose a basis for \mathbb{R}^n and add additional vectors as needed, and we can reach all points, and therefore all 2^n = C(n,d) sign patterns.

For the general case (0), let v_1,\ldots,v_d be vectors in \mathbb{R}^n such that the affine hull with the origin, S = \mathrm{aff}\{0,v_1,\ldots,v_d\}, gives a hyperplane in \mathbb{R}^n that properly intersects the maximum number of orthants, C(n,d). Without loss of generality assume that vectors v_i, i=1,\ldots,d-1 all have their nth component equal to zero. Now, let T = \mathrm{aff}\{0,v_1,\ldots,v_{d-1}\} \subseteq \mathbb{R}^{n-1} be the intersection of S with the (n-1)-dimensional subspace of all points \mathcal{X} = \{x\in\mathbb{R}^n \mid x_n = 0\}, and let C_T denote the number of (n-1)-orthants intersected by T. Note that T itself, as embedded in \mathbb{R}^n, does not properly intersect any orthant. However, by adding or subtracting an arbitrarily small amount of v_d, we intersect 2C_T orthants; taking v_d to be the nth column of the identity matrix would suffice for that matter. Any other orthants that are added have either x_n > 0 or x_n < 0, and their number does not depend on the magnitude of the nth entry of v_d, provided it remains nonzero. Because only the first n-1 entries of v_d determine the maximum number of additional orthants, the problem reduces to \mathbb{R}^{n-1}. In fact, we ask how many new orthants can be added to C_T taking the affine hull of T with v, the orthogonal projection v_d onto \mathcal{X}. Since the maximum orthants for this d-dimensional subspace in \mathbb{R}^{n-1} is given by C(n-1,d), this number is clearly bounded by C(n-1,d)-C_T. Adding this to 2C_T, we have

\begin{aligned} C(n,d) & \leq 2 C_T + [C(n-1,d) - C_T] = C_T + C(n-1,d) \\ & \leq C(n-1,d-1) + C(n-1,d) \\ & \leq 2\sum_{i=0}^{d-1}{n-1 \choose i}. \end{aligned}

The final expression follows by expanding the recurrence relations, which generates (a part of) Pascal’s triangle, and combining this with C(1,j) = 2 for j\geq 1. In the above, whenever there are free orthants in \mathbb{R}^{n-1}, that is, when d < n, we can always choose the corresponding part of v_d in that orthant. As a consequence we have that no hyperplane supported by a set of vectors can intersect the maximum number of orthants when the range of those vectors includes some e_i.

We now show that this expression holds with equality. Let U denote an (n-d)-hyperplane in \mathbb{R}^n that intersects the maximum C(n,n-d) orthants. We now claim that in the interior of each orthant not intersected by U there exists a vector that is orthogonal to U. If this were not the case then T must be aligned with some e_i and can therefore not be optimal. The span of these orthogonal vectors generates a d-hyperplane V that intersects C_V = 2^n - C(n,n-d) orthants, and it follows that

\begin{aligned} C(n,d) & \geq C_V = 2^n - C(n,n-d) \\ & \geq 2^n - 2\sum_{i=0}^{n-d-1}{n-1 \choose i} = 2\sum_{i=0}^{n-1}{n-1 \choose i} - 2\sum_{i=0}^{n-d-1}{n-1 \choose i} \\ & = 2\sum_{n-d}^{n-1}{n-1 \choose i} = 2\sum_{i=0}^{d-1}{n-1 \choose i} \geq C(n,d), \end{aligned}

where the last inequality follows from (?). Consequently, all inequalities hold with equality.

Corollary

Corollary 0

UHJS

Given d \leq n, then C(n,d) = 2^n - C(n,n-d), and C(2d,d) = 2^{2d-1}.

Corollary

Corollary 0

3JRQ

A hyperplane \mathcal{H} in \mathbb{R}^n, defined as the range of V = [v_1,\ v_2, \ldots,\ v_d], intersects the maximum number of orthants C(n,d) whenever \mathop{\hbox{\rm rank}}(V) = n, or when e_i \not\in\mathop{\hbox{\rm range}}(V) for i=1,\ldots,n.

Source and licence

The text on this page is from arXiv:0904.2051v1 by Ewout van den Berg and Michael P. Friedlander, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/0904/0904.2051.jsonl.