An Accelerated Variance-Reduced Conditional Gradient Sliding Algorithm for First-order and Zeroth-order Optimization

We give proof to the first-order case and zeroth-order case respectively.

First-order Case: Lemma 11 implies

Summing the above inequality over s = 1, ..., S, and set x=x^*=\arg\min_{x\in\mathcal{C}}f(x), we have

Next we prove \mathcal{L}_s-\mathcal{R}_{s+1}\leq0 for all s= 1, ..., S-1. When s< s_0, we have \alpha_{s+1}=\alpha_s=\frac{1}{2}, \gamma_{s+1}=\gamma_s, p_{s+1}=p_s=\frac{1}{2}, T_{s+1}=2T_s, thus

When s\geq s_0, we have \alpha_s=\frac{2}{s-s_0+4}, \gamma_s=\frac{1}{3L\alpha_s}, p_{s+1}=p_s=\frac{1}{2}, T_{s+1}=T_s, thus

Thus \mathcal{L}_s-\mathcal{R}_{s+1}\geq0 for s=1, ..., S-1. Note that \mathcal{R}_1=\frac{2}{3L}. Plugging this inequality into (?), we get

where 172 comes from the definition of D_0 that D_0=4(f(\tilde{x}^0)-f(x^*))+3L\|x^0-x^*\|^2.

• If S\leq s_0, then \mathcal{L}_s = \frac{2^{S+1}}{3L}, we have

  \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\leq \frac{D_0}{2^{S+2}}+\frac{3L}{2^{S+1}}\sum_{s=1}^S\sum_{t=1}^{T_s}\eta_{s,t}

  With the choice of \eta_{s,t}, we have

  \sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t} \overset{\textrm{\ding{172}}}{\leq} \sum_{s=1}^{S}\frac{D_0}{sL}\leq\frac{D_0(\log S + 1)}{L}

  where 172 comes from \sum_{k=1}^{n} \frac{1}{k}\leq \log n + 1. Plugging this inequality into (?) we get

  \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\leq \frac{3D_0(\log S + 2)}{2^{S+1}}

• If S>s_0, we have

  \begin{aligned} \mathcal{L}_S=&\frac{1}{3L\alpha_S^2}\left[1+(T_S-1)(\alpha_S+\frac{1}{2})\right]\\ =&\frac{(S-s_0+4)(T_{s_0}-1)}{6L}+\frac{(S-s_0+4)^2(T_{s_0}+1)}{24L} \overset{\textrm{\ding{172}}}{\geq}&\frac{(S-s_0+4)^2n}{48L} \end{aligned}

  where 172 comes from T_{s_0}=2^{\lfloor\log_2n\rfloor+1-1}\geq n/2. Then we have

  \begin{aligned} \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\leq \frac{8D_0}{n(S-s_0+4)^2}+\frac{48L}{n(S-s_0+4)^2}\sum_{s=1}^S\sum_{t=1}^{T_s}\eta_{s,t} \end{aligned}

  With the choice of \eta_{s,t}, we have

  \sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t} \leq \sum_{s=1}^{S}\frac{D_0}{sL}\leq\frac{D_0(\log S + 1)}{L}

  Plugging this inequality into (?) we get

  \begin{aligned} \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\leq\frac{48D_0(\log S + 2)}{n(S-s_0+4)^2} \end{aligned}

  Then we get the desired result for the first-order case.

Zeroth-order Case: From Lemma 11, we get

Summing the above inequality over s=1, ..., S and set x=x^*=\arg\min_{x\in\mathcal{C}}f(x), we have

Next we prove \mathcal{L}_s-\mathcal{R}_{s+1}\leq0 for all s= 1, ..., S-1. When s\leq s_0, we have

When s>s_0, \frac{\gamma_s}{\alpha_s} = \frac{1}{5L\alpha_s^2}= \frac{(s-s_0+4)^2}{20L}, we have

Thus \mathcal{L}_s-\mathcal{R}_{s+1}\geq0 for s=1, ..., S-1. Plugging this inequality into (?), we get

where 172 comes from the definition of D_0 that D_0 = 4(f(\tilde{x}^0)-f(x^*))+5L\|x^0-x^*\|^2.

• If S\leq s_0, then \mathcal{L}_S = \frac{2^{S+1}}{5L}, \alpha_S = \frac{1}{2}, \gamma_S = \frac{2}{5L}. We have

  \begin{aligned} &\mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\\ \leq&\frac{D_0}{2^{S+2}}+\frac{5L}{2^{S+1}}\sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t}+D\mu L\sqrt{d}+3\mu^2Ld\\ \end{aligned}

  With the choice of \eta_{s,t}, we have

  \sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t} \leq \sum_{s=1}^{S}\frac{D_0}{sL}\leq\frac{D_0(\log S + 1)}{L}

  Plugging this inequality into (?) we get

  \begin{aligned} \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right]\leq \frac{5D_0(\log S + 2)}{2^{S+1}}+D\mu L\sqrt{d}+3\mu^2Ld \end{aligned}

• If S>s_0, then \mathcal{L}_s-\mathcal{R}_s = \gamma_sT_s>0 for s>s_0, and \sum_{s=1}^{s_0}2^{s-1} = 2^{s_0}-1\leq 2n. We have

  \begin{aligned} \mathcal{L}_S=&\frac{1}{5L\alpha_S^2}\left[1+(T_S-1)(\alpha_S+\frac{1}{2})\right]\\ =&\frac{(S-s_0+4)(T_{s_0}-1)}{10L}+\frac{(S-s_0+4)^2(T_{s_0}+1)}{40L} \overset{\textrm{\ding{172}}}{\geq}&\frac{(S-s_0+4)^2n}{80L} \end{aligned}

  where 172 comes from T_{s_0}=2^{\lfloor\log_2n\rfloor+1-1}\geq n/2. And

  \begin{aligned} \sum_{s=1}^{S}T_s\frac{\gamma_s}{\alpha_s} =& \sum_{s=1}^{s_0}\frac{2^{s+1}}{5L}+\sum_{s=s_0+1}^S\frac{(s-s_0+4)^2}{20L}2^{s_0-1}\overset{\textrm{\ding{172}}}{\leq} \frac{8n}{5L}+\frac{n(S-s_0+4)^3}{20L} \leq \frac{n(S-s_0+4)^3}{10L} \end{aligned}

  where 172 comes from \sum_{i=1}^ni^2\leq n^3. And

  \begin{aligned} \sum_{s=1}^{S}\gamma_sT_s \leq \frac{1}{2}\sum_{s=1}^{S}T_s\frac{\gamma_s}{\alpha_s}\leq \frac{n(S-s_0+4)^3}{20L} \end{aligned}

  Thus

  \begin{aligned} \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right] \leq&\frac{8D_0}{n(S-s_0+4)^2}+\frac{80L}{n(S-s_0+4)^2}\sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t}+8(S-s_0+4)D\mu L\sqrt{d}\\ &+12(S-s_0+4)\mu^2Ld\\ \end{aligned}

  With the choice of \eta_{s,t}, we have

  \sum_{s=1}^{S}\sum_{t=1}^{T_s}\eta_{s,t} \leq \sum_{s=1}^{S}\frac{D_0}{sL}\leq\frac{D_0(\log S + 1)}{L}

  Plugging this inequality into (?) we get

  \begin{aligned} \mathbb{E}\left[f(\tilde{x}^S)-f(x^*)\right] \leq&\frac{80D_0(\log S + 2)}{n(S-s_0+4)^2}+8(S-s_0+4)D\mu L\sqrt{d} +12(S-s_0+4)\mu^2Ld \end{aligned}

Then we get the desired result for the zeroth-order case. Then we complete the proof.

From Algorithm ?, we have \langle\nabla h(x_t), x_t-x\rangle \leq \eta_{s,t} for any x\in\mathcal{C}. Observe that h is (1+\tau\gamma_s)-strongly convex, then we have

which is

Rearranging the terms, we get

With the notation of l_f(\cdot, \cdot), we have

Also we have

For the term Q_1, we have

With the notation of x_{t-1}^+, we have

which is

Plugging (Proof) into (Proof), we get

Plugging (Proof) and (Proof) into (Proof) and rearranging the terms, we get

Then we complete the proof.

Since f is \tau-strongly convex and L-smooth, then we have

where 172 comes from the definition of \bar{x}_t in Algorithm ?. Plugging Lemma 9 into (Proof), we get

where 172 comes from the fact that f is \tau-strongly convex. Now we give proof to the first-order and zeroth-order case respectively.

First-order Case: Using the fact that b\langle u, v\rangle-a\|v\|^2/2\leq b^2\|u\|^2/(2a), we have from (Proof)

Using Lemma 18, we can bound Q_2 as

where 172 comes from Lemma 18 and 173 comes from the assumption that p_s-\frac{L\alpha_s\gamma_s}{1+\tau\gamma_s-L\alpha_s\gamma_s}>p_s-\frac{2L\alpha_s\gamma_s}{1+\tau\gamma_s-L\alpha_s\gamma_s}>0 and the convexity of f. Plugging (?) into (?), we get

Subtracting both sides with f(x) and then multiplying both sides with \frac{\gamma_s}{\alpha_s}, we get

Then we get the desired result for the first-order case. Next we give proof to the zeroth-order case.

Zeroth-order Case: We can rewrite (Proof) as

where 172 comes from the fact that b\langle u, v\rangle-a\|v\|^2/2\leq b^2\|u\|^2/(2a). Using Lemma 18, we have

where 172 comes from Lemma 18 and 173 comes from the assumption that p_s-\frac{4\alpha_s\gamma_sdL}{1+\tau\gamma_s-L\alpha_s\gamma_s}>0 and the convexity of f. Also we have

which comes from Lemma 18. Plugging (?), (?) into (?), we get

Subtracting both sides with f(x) and then multiplying both sides with \frac{\gamma_s}{\alpha_s}, we get

Then we get the desired result for the zeroth-order case. Then we complete the proof.

Define

From Lemma 10, we have

Summing the above inequality over t= 1, ..., T_s, with the definition of \theta_t, we have

Note that \tilde{x}^s = \sum_{t=1}^{T}(\theta_t\bar{x}_t)/\sum_{t=1}^T\theta_t, x^s = x_{T_s}, x^{s-1} = x_0, \tilde{x}^{s-1}=\tilde{x}, the convexity of f and Jensen’s inequality, we have

Denote \mathcal{L}_s = \frac{\gamma_s}{\alpha_s}+(T_s-1)\frac{\gamma_s(\alpha_s+p_s)}{\alpha_s}, \mathcal{R}_s = \frac{\gamma_s}{\alpha_s}(1-\alpha_s)+(T_s-1)\frac{\gamma_sp_s}{\alpha_s}, we have

First-order Case: Using the definition of \Delta_t and (Proof), we have

Then we get the desired result for first-order case.

Zeroth-order Case: Using the definition of \Delta_t and (Proof), we have

Next we have

where 172 comes from Cauchy-Schwartz inequality and 173 comes from Lemma 16 and Assumption 2. Plugging (?) into (?), we get

Then we get the desired result for zeroth-order case. Then we complete the proof.

For the first-order case, the result is the same as that in Lemma 10. Now we give proof to the result of the zeroth-order case. From Lemma 10 we have

From b\langle u, v\rangle-\frac{a}{2}\|v\|^2\leq\frac{b^2}{2a}\|u\|^2 we have for c>0

Plugging (Proof) into (Proof), we get

where 172 comes from Lemma 16. Then we complete the proof.

We give proof to the first-order case and zeroth-order case respectively.

First-order Case: We have \alpha_s=p_s=\frac{1}{2}, \gamma_s=\frac{2}{3L}, T_s=2^{s-1}. Summing up Lemma 12 for t=1, ..., T_s, we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1}, the fact that \frac{\gamma_s}{\alpha_s}=\frac{4}{3L}, the convexity of f and Jensen’s inequality, we have

Summing up (?) for s= 1, ..., S, we get

Set x=x^*, we get

With the choice of \eta_{s,t}, we have

Plugging this inequality into (?), with the definition of D_0 and T_s=2^{s-1}, we get

Then we get the desired result for the first-order case.

Zeroth-order Case: We have \alpha_s=p_s=\frac{1}{2}, \gamma_s=\frac{2}{5L}, T_s=2^{s-1}. Summing up Lemma 12 with c=1 for t=1, ..., T_s, we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1}, the fact that \frac{\gamma_s}{\alpha_s}=\frac{4}{5L} , the convexity of f and Jensen’s inequality, we have

Summing up (?) for s=1, ..., S, we get

Note that T_s=2^{s-1}. Setting x=x^*, we have

where 172 comes from the definition of D_0. With the choice of \eta_{s,t}, we have

Plugging this inequality into (?) we get

Then we get the desired result for the zeroth-order case. Then we complete the proof.

We give proof to the first-order case and zeroth-order case respectively.

First-order Case: We have \alpha_s=p_s=\frac{1}{2}, \gamma_s = \frac{2}{3L}. Note that for the first-order case, we have f=f. From Lemma 12 we have

Multiplying both sides of (Proof) with \theta_t=\Gamma_{t-1}=\left(1+\tau\gamma_s\right)^{t-1}, we get

Summing up (?) for t=1, ..., T_s, we get

Since we have

where 172 comes from the assumption that n\geq\frac{3L}{4\tau}. Then we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1}, the fact that \frac{\gamma_s}{\alpha_s}=\frac{4}{3L}, the convexity of f and Jensen’s inequality, we have

Applying the inequality recursively for s\geq s_0, we get

where 172 comes from the choice of \eta_{s,t} that

and \sum_{t=1}^{T_s}\theta_t\geq T_s=T_{s_0}. From (?) we have

Plugging (?) into (?), setting x=x^*, we get

where 172 comes from the fact that 2^{s_0}\geq n. Then we get the desired result for the first-order case.

Zeroth-order Case: We have \alpha_s=p_s=\frac{1}{2}, \gamma_s=\frac{2}{5L}, T_s=T_{s_0}=2^{s_0-1}. Setting c=\frac{1}{2} in Lemma 12, we have

Multiplying both sides of (?) with \theta_t=\Gamma_{t-1}=\left(1+\frac{\tau\gamma_s}{2}\right)^{t-1}, we get

Summing up (?) for t=1, ..., T_s, we get

Since we have

where 172 comes from the assumption that n\geq\frac{5L}{4\tau}. Then we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1}, the fact that \frac{\gamma_s}{\alpha_s}=\frac{4}{5L}, the convexity of f and Jensen’s inequality, we have

Applying the inequality recursively for s\geq s_0, we get

where 172 comes from the choice of \eta_{s,t} that

and

and \sum_{t=1}^{T_s}\theta_t\geq T_s=T_{s_0}. From (?) we have

Plugging (?) into (?), setting x=x^*, we get

where 172 comes from the fact that 2^{s_0}\geq n. Then we get the desired result for the zeroth-order case. Then we complete the proof.

We give proof to the first-order case and zeroth-order case respectively.

First-order Case: We have \alpha_s=\sqrt{\frac{n\tau}{3L}}, p_s = \frac{1}{2}, \gamma_s=\frac{1}{\sqrt{3n\tau L}}, T_s=T_{s_0}=2^{s_0-1}. Note that for the first-order case, we have f=f. From Lemma 12, we have

Multiplying both sides of the above inequality with \Gamma_{t-1}=\left(1+\tau\gamma_s\right)^{t-1}, we get

Summing up the above inequality for t=1, ..., T_s, using the definition of \theta_t, we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1}, the convexity of f and Jensen’s inequality, we have

From the definition of \theta_t, we have

Since T_{s_0}=2^{s_0-1}\leq n, we have

Then we have

where 172 comes from (?) and 173 comes from the fact that (1+a)^b\leq 1+2ab, for b\geq1, ab\in[0, 1]. Plugging (?) into (?), we get

Plugging (?) into (?), we get

Since we have

Dividing both sides of the above inequality with \Gamma_{T_{s_0}}\frac{\gamma_s}{\alpha_s}\left[1-\alpha_s-p_s+p_s\sum_{t=1}^{T_{s_0}}\Gamma_{t-1}\right], we have

Summing up the above inequality for s>s_0, setting x=x^*, we get

where 172 comes from the fact that \frac{\alpha_s}{\gamma_s}=n\tau\leq\frac{3L}{4}. From Theorem 13 we have

where 172 comes from (?). With the choice of \eta_{s,t}, we have

Plugging (?), (?) into (?), we get

where 172 comes from 2^{s_0}\geq n. From the definition of \Gamma_{T_{s_0}}, we have

Plugging (?) into (?), we get

Then we get the desired result for the first-order case.

Zeroth-order Case: We have \alpha_s=\sqrt{\frac{n\tau}{5L}}, p_s=\frac{1}{2}, \gamma_s=\frac{1}{\sqrt{5n\tau L}}, T_s=T_{s_0}=2^{s_0-1}. Setting c=\frac{1}{2} in Lemma 12, we have

Multiplying both sides of the inequality with \Gamma_{t-1}=\left(1+\frac{\tau\gamma_s}{2}\right)^{t-1}, we get

Summing up the inequality for t=1, ..., T_s, using the definition of \theta_t, we get

From the definition of \tilde{x}^s, \tilde{x}^{s-1}, x^s, x^{s-1} and the convexity of f, we have

From the definition of \theta_t, we have

Since T_{s_0}=2^{s_0-1}\leq n, we have

Then we have

where 172 comes from (?) and 173 comes from the fact that (1+a)^b\leq1+2ab, for b\geq1, ab\in[0, 1]. Plugging (?) into (?), we get

Then plugging (?) into (?), setting x=x^*, we get

Denote \alpha=\alpha_s=\sqrt{\frac{n\tau}{5L}}, p=p_s=\frac{1}{2}, \gamma=\gamma_s=\frac{1}{\sqrt{5n\tau L}} Rearranging the terms and summing up the above inequality for s>s_0, we get

Since we have

Plugging (?) into (?), we get

where 172 comes from the choice of \eta_{s,t} that

and

From (?) we know

From the definition of \alpha, \gamma and the assumption that n<\frac{5L}{4\tau}, we have \frac{\alpha}{\gamma}=5L\alpha^2=n\tau\leq \frac{5L}{4}. Thus we get

where 172 holds since 2^{s_0}\geq n. Plugging (?) into (?), we get

From the definition of \Gamma_{T_{s_0}}, we have

where 172 comes from the definition of \gamma. Then we have

Plugging (?) into (?), using the definition of \alpha, \gamma, we get

Then we get the desired result for the zeroth-order case. Then we complete the proof.

See [[ji2019improved], Appendix, Lemma 3].

Denote \phi_i(x)= f_i(x)-f(y)-\langle\nabla f(y), x-y\rangle. It is easy to verify that \phi_i is also L-smooth. Clearly \nabla \phi_i(y) = 0 and hence \min_{x\in\mathcal{C}}\phi_i(x) = \phi_i(y) = 0. Then for \alpha\in\mathbb{R}, we have

where 172 comes from the smoothness of \phi_i. Rearranging the terms and using the definition of \phi_i we get

Taking expectation with respect to i, we get

Then we complete the proof.

The part for the first-order case is proved in [[lan2019unified], Lemma 3]. Now we give a proof to the zeroth-order case. For the zeroth-order case, we have

Then we prove the upper bound of \mathbb{E}\left[\|\delta_t-\mathbb{E}\left[\delta_t\right]\|^2\right]. We have

where 172 comes from \mathbb{E}\left[\|x-\mathbb{E}\left[x\right]\|^2\right]=\mathbb{E}\left[\|x\|^2\right]-\mathbb{E}\left[x\right]^2\leq\mathbb{E}\left[\|x\|^2\right], 173 comes from the Cauchy-Schwarz inequality, 174 comes from \mathbb{E}\left[\|x-\mathbb{E}\left[x\right]\|^2\right]\leq\mathbb{E}\left[\|x\|^2\right] and Lemma 16, 175 comes from Lemma 17. Then we complete the proof.

Since f is L-smooth, then we have

where 172 comes from the definition of \underline{x}_t and x_t, 173 comes from the convexity of f, 174 comes from Line ? of Algorithm ?, 175 comes from the fact that b\langle u, v\rangle-a\|v\|^2/2\leq b^2\|u\|^2/(2a). Note that \mathbb{E}\left[\langle \delta_t, x^*-x_{t-1}\rangle\right] = 0. So with the condition \mathbb{E}\left[\|\delta_t\|^2\|\right]\leq\frac{L^2D_s^2}{T_s(k+1)^2} \overset{\textrm{def}}{=} \sigma_t^2 we arrive at

Now we define \Gamma_{t} = \Gamma_{t-1}\left(1-\alpha_{s,t}\right) when t>1 and \Gamma_1 = 1. By induction, one can verify \Gamma_{t} = \frac{2}{t(t+1)} and the following:

which is at most

Finally plugging in the parameters \alpha_{s,k}, \gamma_{s,k}, \eta_{s,k}, \Gamma_k and the bound \mathbb{E}\left[\|\bar{x}_0-x^*\|^2\right] \leq D_s^2 concludes the proof: