On Subspace Approximation and Subset Selection in Fewer Passes by MCMC Sampling

Consider the l=1, t=1 case of the above MCMC sampling procedure. In this case, the procedure outputs only one element of \mathcal{X}. This random element is picked by m steps of the following random walk. We first pick x with probability q(x), then we sample a y with probability q(y), we compute p(x), p(y) and sample a number uniformly at random from the interval (0, 1), and if p(y) q(x)/p(x) q(y) > \mathrm{Unif}(0,1), then the random walk moves from x to y and updates p(x) as p(y). Observe that the stationary distribution of the above random walk is the adaptive sampling distribution w.r.t. S given by p(x) = d(x, \operatorname{span}\left(S\right))^{p}/\mathrm{err}_{p}(\mathcal{X}, S). Using Corollary 1 of [cai], the total variation distance after m steps of the random walk is bounded by

The above bound is at most \epsilon_{2} if we choose to run the random walk for m \geq 1+\gamma \log \frac{1}{\epsilon_2} steps. Now suppose \mathrm{err}_{p}(\mathcal{X}, S) > \epsilon_{1}~ \mathrm{err}_{p}(\mathcal{X}, \tilde{S}). Then, for any x \in \mathcal{X}

using \tilde{S} \subseteq S and the above assumption \mathrm{err}_{p}(\mathcal{X}, S) > \epsilon_{1}~ \mathrm{err}_{p}(\mathcal{X}, \tilde{S}). Therefore, m > \frac{2}{\epsilon_{1}} \log\frac{1}{\epsilon_{2}} ensures that m steps of the random walk gives a distribution within total variation distance \epsilon_{2} from the adaptive sampling distribution for picking a single point.

Note that for t > 1 both the adaptive sampling and the MCMC sampling procedure pick an i.i.d. sample of t points, so the total variation distance is additive in t, which means

assuming \mathrm{err}_{p}(\mathcal{X}, S) > \epsilon_{1}~ \mathrm{err}_{p}(\mathcal{X}, \tilde{S}). This completes the proof the lemma.

We show a slightly stronger inequality than the one given above, i.e., for any S_{0} such that \tilde{S} \subseteq S_{0},

The special case S_{0} = \tilde{S} gives the lemma. We prove the above-mentioned stronger statement by induction on l. For l=0, the above inequality holds trivially. Now assuming induction hypothesis, the above holds true for l-1 iterations (instead of l) starting with any subset S_{1} = S_{0} \cup A \subseteq \mathcal{X} because \tilde{S} \subseteq S_{0} \subseteq S_{1}.

If \mathrm{ind}_{p}(\mathcal{X}, S_{1}) = 1 then \mathrm{err}_{p}(\mathcal{X}, S_{1} \cup T_{2:l}) \leq \mathrm{err}_{p}(\mathcal{X}, S_{1}) \leq \epsilon_{1}~ \mathrm{err}_{p}(\mathcal{X}, S_{0}), so the first part of the above sum can be bounded as follows.

Now, the second part can be bounded as follows.

using Lemma 4.1 about the total variation distance between P^{(1)} and \tilde{P}^{(1)} distributions. Plugging the bounds \eqref{eq:part-1-ind} and \eqref{eq:part-2-ind} into \eqref{eq:parts-by-ind}, we get

which completes the proof of Lemma 4.2.

Follows directly from Corollary 1 in [DBLP:conf/approx/DeshpandeV06].

We use the following setting of parameters t, l, \epsilon_{1}, \epsilon_{2}, m as functions of k, \epsilon, \alpha.

We have

Anari et al. [anari2016monte] give an MCMC algorithm to sample approximately from volume sampling. They start with any subset S, pick i \in S and j \notin S uniformly at random, and perform a lazy random walk over k-sized subsets, i.e., move to T = S \setminus \{i\} \cup \{j\} with probability \frac{1}{2} \min\{1, \frac{\mathrm{vol}(\Delta_{T})^{2}}{\mathrm{vol}(\Delta_{S})^{2}}\}, and with the remaining probability stay at S. They show that in \text{poly}(n, k, \log(1/\delta \mathrm{vol}(\Delta_{S})) steps of the above random walk starting at S, the resulting distribution is within \delta total variation distance from the exact volume sampling distribution. Thus, using Proposition 4.4 with \alpha = 1 and adding the error due to total variation distance, we get that the expected error \mathrm{err}_{2}(\mathcal{X}, S) at the end of \text{poly}(n, k, \log(1/\delta \mathrm{vol}(\Delta_{S})) steps of the above random walk is at most (k+1)~ \mathrm{err}_{p}(\mathcal{X}, V^{*}) + \delta \max_{S \;:\; \left|S\right|=k} \mathrm{err}_{p}(\mathcal{X}, S), where V^{*} minimizes \mathrm{err}_{2}(\mathcal{X}, V) over all linear subspaces V of dimension k. We choose \delta as follows.

where \sigma_{\min} and \sigma_{\max} are the minimum and the maximum singular values, respectively, of the n-by-d matrix whose rows are all the points in \mathcal{X}, and \kappa is the condition number. Then the expected error of \mathrm{err}_{2}(\mathcal{X}, S) is at most (k+2)~ \mathrm{err}_{p}(\mathcal{X}, V^{*}). Observe that \min_{S \;:\; \left|S\right|=k} \mathrm{vol}(\Delta_{S})^{2} \geq \sigma_{\min}^{k}. Note that for the above choice of \delta and any k-sized subset S \subseteq \mathcal{X}, we can upper bound \log(1/\delta \mathrm{vol}(\Delta_{S})^{2}) by O(k \log \kappa), where \kappa is the condition number of the n-by-d matrix whose rows are the points of \mathcal{X}. Thus, in total number of \text{poly}(n, k, \log \kappa) steps of the random walk we get an initial subset S_{0} with \mathrm{err}_{2}(\mathcal{X}, S_{0}) upper bounded by (k+2)~ \mathrm{err}_{2}(\mathcal{X}, S^{*}), in expectation.

Another alternative to implement approximate volume sampling is to do a (1+\epsilon) volume-preseving random projection using only one pass over the given data [DBLP:conf/approx/MagenZ08], and then do exact volume sampling on the projected data in \mathrm{poly}(k/\epsilon) dimensions. This gives another alternative to obtain the initial subset S_{0}.

The rest of the proof using MCMC sampling algorithm starting from S_{0} using the same subset as a pivot is similar to the proof of Theorem 4.5.

The running time of MCMC procedure is O(tlm \cdot d(tl)^2)=d\cdot\mathrm{poly}(k/\epsilon), where d\cdot(tl)^2 is the time required to project a point onto \mathrm{span}(S), where S is a subset of O(tl) points of dimension d each.

Follows from Theorems 11, 12 of [DBLP:journals/tcs/DeshpandeP21] and Theorem 4.5 of this paper.