The Geometry and Well-Posedness of Sparse Regularized Linear Regression

Let A\in \mathbb{R}^{m\times n} be a linear operator and f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function, i.e., f(\bm{x}) = \max_{i = 1, \dots k}\{\bm{v}_i^T\bm{x} + w_i\} + \chi_{\mathbf{P}}(\bm{x}) with \bm{v}_i \in \mathbb{R}^n, w_i \in \mathbb{R}, \mathbf{P} a polyhedral set and \chi_{\mathbf{P}}(\bm{x}) is zero if \bm{x} \in \mathbf{P} and infinite otherwise. Furthermore, assume that \dim(\operatorname{dom}(f)) = n, where \operatorname{dom}(f) is the effective domain of f, i.e., the points \bm{x} in the domain for which f(\bm{x}) is finite. Then, the sparse regularized linear least squares problem

is well-posed if and only if \operatorname{row}(A) \cap \partial f(\bm{x}) \neq \emptyset for \bm{x} \in \mathbb{R}^n implies that \dim(\partial f(\bm{x})) \geq \operatorname{nullity}(A), where \operatorname{row}(A) is the row space of A, \operatorname{nullity}(A) is the dimension of the null space of A and \partial f(\bm{x}) is the subdifferential of f at \bm{x}.

Let f : \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. Each polyhedral set \mathbf{F} \in \mathcal{F}_f can be characterized as follows. Let \bm{x} \in \operatorname{relint}(\mathbf{F}) be arbitrary, then the relative interior \operatorname{relint}(\mathbf{F}) consists precisely of all \bm{y}\in \mathbb{R}^n for which \partial f(\bm{x}) = \partial f(\bm{y}).

The constant subdifferential corresponding to \operatorname{relint}(\mathbf{F}) will be denoted by \partial f_{\mathbf{F}}.

Let f : \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. Let \mathbf{F}, \mathbf{H} \in \mathcal{F}_f with corresponding \partial f_{\mathbf{F}}, \partial f_{\mathbf{H}} \in \mathcal{F}_f^\star, then \mathbf{F} is a face of \mathbf{H} if and only if \partial f_\mathbf{H} is a face of \partial f_{\mathbf{F}}.

Let f : \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. Then,

Let f : \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. Let \mathbf{F} \in \mathcal{F}_f with corresponding \partial f_{\mathbf{F}} \in \mathcal{F}_f^\star, then the following holds:

40 \dim(\mathbf{F}) + \dim(\partial f_{\mathbf{F}}) = n . 40 \partial f_{\mathbf{F}} \bot \mathbf{F}, i.e., (\partial f_{\mathbf{F}} - \partial f_{\mathbf{F}})^T(\mathbf{F} - \mathbf{F}) = \{\bm{0}\}. 40 \mathbf{F} + \partial f_{\mathbf{F}}^\bot \in \operatorname{aff}(\mathbf{F}).

The vertices of the polytope \mathbf{P}_{\text{TV}_G} are the vectors \bm{z}_{\bm{u}} = \sum_{\{v_i, v_j\} \in \mathbf{E}} u_{i,j}\bm{d}_{i,j} with \bm{u} \in \{-1,1\}^{\#\mathbf{E}} such that the associated directed graph is acyclic.

An undirected graph G is a tree, i.e., the graph does not have cycles, if and only if, the vertices of \mathbf{P}_{\text{TV}_G} are all the vectors of the form \sum_{\{v_i, v_j\} \in \mathbf{E}}u_{i,j}\bm{d}_{i,j} with \bm{u} \in \{-1,1\}^{\#\mathbf{E}}.

The vertices of \mathbf{P}_{\text{TV}_G} - \mathbb{R}_{\geq 0}^{\#\mathbf{V}} and \mathbf{P}_{\text{TV}_G} are the same.

Let f: \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. If \mathbf{L} \in \mathcal{L}_f(c) for some c\in\mathbb{R} and \bm{x} \in \operatorname{relint}(\mathbf{L}) and \bm{x} \in \mathbf{F} \in \mathcal{F}, then \mathbf{L} \subseteq \mathbf{F}.

Let f: \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. For every \mathbf{L} \in \mathcal{L}_f(c) with c\in\mathbb{R}, let \bm{x}\in\operatorname{relint}(\mathbf{L}), then \partial f(\bm{x})\bot \mathbf{L}, i.e., \partial f(\bm{x})^T(\mathbf{L}-\mathbf{L}) = \{\bm{0}\}.

Particularly, \mathbf{L} + \partial f(\bm{x})^\bot \in \operatorname{aff}(\mathbf{L}).

Let f: \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear. Let \partial f_{\mathbf{F}} \in \mathcal{F}_f^\star. If \operatorname{codim}(\partial f_{\mathbf{F}}) - \operatorname{rank}(A) = k > 0 then \dim(\operatorname{Null}(A) \cap \partial f_\mathbf{F}^\bot) = k.

Let f: \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear, A \in \mathbb{R}^{m\times n} and \bm{x} \in \mathbf{F} \in \mathcal{F}_f with \operatorname{codim}(\partial f_{\mathbf{F}}) - \operatorname{rank}(A) = k > 0. Then there exists \mathbf{L} \in \mathcal{L}_f(f(\bm{x})) and a polyhedral set \mathbf{S} \subseteq \mathbf{L} \subseteq \mathbf{F} for which \bm{x} \in \mathbf{S}, \dim(\mathbf{S}) = k and \mathbf{S} - \mathbf{S} \subseteq \operatorname{Null}(A).

Let f: \mathbb{R}^{n} \rightarrow \mathbb{R} \cup \{\infty\} be convex piecewise linear such that \dim(\operatorname{dom}(f)) = n and A \in \mathbb{R}^{m\times n}. If there exist distinct \bm{x}, \bm{y} \in \mathbb{R}^n such that \bm{x} - \bm{y} \in \operatorname{Null}(A) and \operatorname{row}(A) \cap \partial f(\bm{x}) \cap \partial f(\bm{y}) \neq \emptyset, then there exist \mathbf{F} \in \mathcal{F}_f such that \bm{x},\bm{y}\in \mathbf{F} and \operatorname{codim}(\partial f_{\mathbf{F}}) > \operatorname{rank}(A).

Consider the Bregman divergence associated to a continuously differentiable convex function \phi: \mathbb{R}^m \rightarrow \mathbb{R}, i.e., D_\phi(\bm{z}, \bm{b}) = \phi(\bm{z}) - \phi(\bm{b}) - \nabla \phi(\bm{b})^T(\bm{z} - \bm{b}). Assume that \phi is strictly convex and \operatorname{range}(\nabla \phi) = \mathbb{R}^m, then D_\phi is a smooth data fidelity function.

Let \psi : \mathbb{R}^m \rightarrow \mathbb{R} be differentiable and strictly convex such that \nabla \psi : \mathbb{R}^m \rightarrow \mathbb{R}^m is bijective, then D(\bm{z}, \bm{b}) = \psi(\bm{z} - \bm{b}) is a smooth data fidelity function.

Let f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a proper convex lower semi-continuous function and D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} a smooth data fidelity function. Then,

Let f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function and D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} a smooth data fidelity function. For any \mathbf{F} \in \mathcal{F}_f we have

Let f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function and D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} a smooth data fidelity function. Then, there exists a polyhedral subcomplex \mathcal{E} \subseteq \mathcal{F}_f, such that

Let f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a proper convex lower semi-continuous function and let D: \mathbb{R}^m\times \mathbb{R}^m \rightarrow \mathbb{R} be strictly convex in its first argument. For every \bm{b} \in \mathbb{R}^m, if \bm{x}, \bm{y} \in \mathbf{S}^D_{A,f}(\bm{b}), then A\bm{x} = A\bm{y} and f(\bm{x}) = f(\bm{y}).

(Necessary condition for uniqueness) Let D:\mathbb{R}^m\times\mathbb{R}^m \rightarrow \mathbb{R} be arbitrary. If there exist \bm{x} \in \mathbf{S}_{A, f}^{D}(\bm{b}) for some \bm{b} \in \mathbb{R}^m and let \mathbf{F} \in \mathcal{F}_f containing \bm{x} such that \operatorname{codim}(\partial f_{\mathbf{F}}) - \operatorname{rank}(A) = k > 0, then \dim(\mathbf{S}_{A, f}^{D}(\bm{b})) \geq k.

Thus, if \#[\mathbf{F} \cap \mathbf{S}_{A, f}^{D}(\bm{b})] \leq 1 for all \bm{b} \in \mathbb{R}^m and \mathbf{F} \in \mathcal{F}_f, then \#[\mathbf{F} \cap \mathbf{S}_{A, f}^{D}(\bm{b})] = 1 must imply \operatorname{codim}(\partial f_{\mathbf{F}}) \leq \operatorname{rank}(A).

Let D: \mathbb{R}^m\times\mathbb{R}^m \rightarrow \mathbb{R} be strictly convex and differentiable in its first argument and \dim(\operatorname{dom}(f)) = n. If \#\mathbf{S}_{A, f}^{D}(\bm{b}) > 1 for some \bm{b} \in \mathbb{R}^m, then there exists \mathbf{F} \in \mathcal{F}_f such that \operatorname{row}(A) \cap \partial f_{\mathbf{F}} and \operatorname{codim}(\partial f_{\mathbf{F}}) > \operatorname{rank}(A).

Let D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} be a smooth data fidelity function, A\in \mathbb{R}^{m\times n} be a linear operator and f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function. If \dim(\operatorname{dom}(f)) = n, then \#\mathbf{S}_{A,f}^D(\bm{b}) = 1 for all \bm{b}\in\mathbb{R}^m if and only if \operatorname{row}(A) \cap \partial f_{\mathbf{F}} \neq \emptyset implies \operatorname{codim}(\partial f_{\mathbf{F}}) \leq \operatorname{rank}(A). That is, every accessible set \mathbf{F} must satisfy \operatorname{codim}(\partial f_{\mathbf{F}}) \leq \operatorname{rank}(A).

Let D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} be D(\bm{x}, \bm{y}) = \frac12\|\bm{x} - \bm{y}\|^2_{\Sigma}, A \in \mathbb{R}^{m\times n} be a linear operator and f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function. If \mathbf{S}^D_{A, f} is single-valued, then for all \mathbf{F} \in \mathcal{F}_f, the inverse image [S^D_{A,f}]^{-1}(\mathbf{F}) is closed and convex and the restricted solution map S^D_{A, f}\big|_{[S^D_{A,f}]^{-1}(\mathbf{F})} is affine.

Let D : \mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R} be D(\bm{x}, \bm{y}) = \|\bm{x} - \bm{y}\|^2_{\Sigma}, A \in \mathbb{R}^{m\times n} be a linear operator and f:\mathbb{R}^n \rightarrow \mathbb{R} \cup \{\infty\} be a convex piecewise linear function. If \mathbf{S}_{A, f} is single-valued, then S_{A,f} is continuous.

Let f : \mathbb{R}^n \rightarrow \cup \{\infty\} be a convex piecewise linear function with \dim(\operatorname{dom}(f)) = n and p be the dimension of the smallest subdifferential \partial f_{\mathbf{F}} \in \mathcal{F}_f^{\star} that contains the zero vector. If A \in \mathbb{R}^{m \times n} with m \geq p is a continuously distributed random matrix, then the linear least squares problem for A regularized by f is almost surely well-posed.

For any B\in \mathbb{R}^{k\times n} and \bm{y} \in \mathbb{R}^{k}, there exists A \in \mathbb{R}^{m\times n} and f(\bm{x}) = \chi_{\|\cdot\|_\infty \leq 1}(\bm{x}) - \bm{v}^T\bm{x} for some \bm{v} \in \mathbb{R}^n, such that the l_0 minimization problem

is equivalent to

Similarly, the nonnegative l_0 minimization problem

can be transformed into (10) for f(\bm{x}) = \chi_{\bm{v} - \mathbb{R}^{n}_{\geq 0}}(\bm{x}) - \bm{v}^T\bm{x} for some \bm{v} \in \mathbb{R}^n.

Solving

for convex piecewise linear functions f is NP-hard.

Let f_n(\bm{x}) be either \gamma \|\bm{x}\|_1, \gamma \|D_n\bm{x}\|_1 or \gamma \|D_n\bm{x}\|_1 + \chi_{\mathbb{R}_{\geq 0}^n} with finite difference matrix D_{n} \in \mathbb{R}^{(n-1)\times n} and \gamma > 0. Then, verifying well-posedness of A\in \mathbb{R}^{m\times n} with respect to f_n(\bm{x}) is co-NP-hard.