On the Complexity of Finding Stationary Points in Stochastic (L_0, L_1)-Smooth Nonconvex Optimization

Recall that the functions \psi and \phi are defined as

respectively, where

We verify the first-order and second-order derivatives of S as

and

We now prove each statement of the lemma.

The proof of statement (a). The definitions of \psi and S in equations (Proof) and (Proof) directly shows that \psi(t)=0 for all t \le 1/2, and \psi(t)=1 for all t\ge1.

The proof of statement (b). We first consider the value of \psi(t). By equation equation (Proof), we know that S'(t)\geq 0 on {\mathbb{R}}, which implies the function S is non-decreasing. Combining the facts S(0)=0 and S(1)=1, we obtain 0\le S(u)\le1 for all u\in[0,1]. Combing the definitions S(u)=0 for all u\leq 0 and S(u)=1 for all u\geq 1, we know that S(u)\in[0,1] on {\mathbb{R}}. Hence, the definition of \psi(t)=S(2t-1) implies

We then consider the value of \psi'(t). By the chain rule, we can write

Clearly, it holds \psi'(t)\ge0. For t\in[1/2,1], we let

which leads to

Since for all u\in[0,1], it holds

which implies

Recall that \psi'(t)=0 for all t\in[0, 1/2], we obtain 0\le\psi'(t)\le4 for all t\in{\mathbb{R}}.

We now consider the value of \psi''(t). By the chain rule, we can write

Equation (Proof) indicates \psi''(t)=0 for all t\not\in(1/2,1). Hence, we focus on the case of t\in(1/2,1), that is u=2t-1\in(0,1). We write

and define

Then we have

which has critical points

Since it holds g(0)=g(1)=0, the maximum of |g(u)| on [0,1] is attained at one of these critical points. We can verify

which implies

The proof of statement (c). The fact \tanh t\in(-1,1) directly implies

We can also verify \phi'(t)=\operatorname{sech}^2t and \phi''(t) = -2\operatorname{sech}^2(t)\tanh(t), which implies

The proof of statement (d). The statement that \phi' is even can be verifies as

It remains to show that \phi'(t)>2/5 for |t|\le1. For all t\geq 0, it holds

which implies \phi' is non-increasing on [0,\infty). Therefore, for all t\in[0,1], it holds

Since \phi'(t) is even, we have |\phi'(t)|>2/5 when |t|\leq 1.

Recall that the function \bar f_T is defined as

The proof of statement (a). According to definition (Proof) and Lemma 3.1(a), we have \phi(0)=1 and \psi(0)=0. Hence, definition (Proof) implies

For all {\bf{y}}\in{\mathbb{R}}^T, Lemma 3.1(c) implies

Moreover, for every i\ge2, Lemma 3.1(b) and (c) implies

It follows from equation (Proof) that

Thus, it holds

We then consider the vector

Lemma 3.1(a) implies \psi(r)=1 and \psi(-r)=0. Consequently, definition equation (Proof) implies

for all i\in[T], which results

Combining equations (?) and (?) yields

The proof of statement (b). We first establish the uniform bound on the gradient of \bar f_T. Taking the derivative on equation (Proof) and using the evenness of \phi' from Lemma 3.1(d), we obtain

According to Lemma 3.1(b) and (c) and Lemma A.1, we have

Combing with equation (?) implies

We now consider the bound of Hessian. Lemma 3.1(c) implies

For all i\ge2, Lemma 3.1(c) and A.1 imply

Note that each A_i({\bf{y}}) depends only on y_{i-1} and y_i for i\ge2, which implies the Hessian of \bar f_T({\bf{y}}) is tridiagonal matrix, i.e.,

According to equations (?) and (?), the absolute value of each diagonal entry of \nabla^2\bar f_T({\bf{y}}) is bounded by 48+2=50. Moreover, equation (?) implies that each nonzero off-diagonal entry of \nabla^2\bar f_T({\bf{y}}) has absolute value at most 4. Hence, every row of \nabla^2\bar f_T({\bf{y}}) has absolute row sum at most 50+2\cdot4=58. Since the Hessian is symmetric, its spectral norm is bounded by its maximum absolute row sum, which implies

The proof of statement (c). We let i\in[T] be the smallest index such that |y_i|\le1. If i=1, then equation (?) and the nonnegativity of \psi' and \phi from Lemma 3.1(b) and (c) implies

Combining with |y_1|\le1 and Lemma 3.1(d), we achieve

If i\ge2, the definition of i implies |y_{i-1}|>1. According to Lemma 3.1(a), we obtain

Substituting equation (?) into equation (?) and using the non-negativity of \psi' and \phi from Lemma 3.1(b) and (c), we have

Since |y_i|\le1, Lemma 3.1(d) implies

Combing above two inequality, we obtain

The proof of statement (d). We now establish the gradient-dependent upper bound for Hessian. We first consider the case that there exists some i\in[T] such that |y_i|\le1. By equation (?) and equation (?), we obtain

We then consider the case that |y_i|>1 holds for all i\in[T]. According to Lemma 3.1(a), we have

Moreover, since both y_i and -y_i do not lie in the interval (1/2,1), equation (Proof) implies

Substituting the above two identities into equation (?), we obtain

Hence, the Hessian of \bar f_T({\bf{y}}) is diagonal and satisfies

Consequently, Lemma 3.1(c) and equation (?) imply

Combining equations (?) and (?), we conclude that

for all {\bf{y}}\in{\mathbb{R}}^T.

The proof of statement (e). Follow the notation defined in Section ?, we let

for given {\bf{y}}=[y_1, \dots, y_T]\in{\mathbb{R}}^T. For each i>\operatorname{prog}_{1/4}({\bf{y}})+1, we have

Combining Lemma 3.1(a) and equation equation (Proof), equation (?) implies

Substituting the above results into equation (?), we obtain

for all i>\operatorname{prog}_{1/4}({\bf{y}})+1. Hence, it holds

Recall that equation (?) has defined

where {\bf U}=[{\bf{u}}_1,\ldots,{\bf{u}}_T]\in\operatorname{St}(d,T). Throughout the proof of this lemma, we denote

and recall from equation (?) that \rho_R({\bf{x}})=\alpha({\bf{x}}){\bf{x}}. By Lemma 3.2(b), we have

The optimality gap. Note that equation (?) implies \rho_R({\bf{0}})={\bf{0}}. Combining with equation (?), we obtain

Moreover, the lower bound of \bar f_T({\bf{y}})\ge-2T in equation (?) implies

Combining above two results, we have

The lower bound of the Hessian. According to the chain rule and equation (?), the gradient of H_{{\bf U},\mu} has the form of

Since we have assumed there exists some i\in[T] such that |y_i|=\left| [{\bf U}^\top\rho_R({\bf{x}})]_i\right|\le1, we denote

If i_\star=1, then equation (?) and the non-negativity of \psi' and \phi from Lemma 3.1(b) and (c) imply

If i_\star\ge2, the minimality of i_\star gives |y_{i_\star-1}|>1. Hence, Lemma 3.1(a) implies

Substituting the above result into equation (?) also yields equation (?). Thus, Lemma 3.1(d) implies

In the case of \left\| {\bf{x}} \right\|\ge20\sqrt T/\mu, we apply equation (?), (Proof), and Proof to achieve

In the case of \left\| {\bf{x}} \right\|<20\sqrt T/\mu. The setting R=1000\sqrt T/\mu implies \left\| {\bf{x}} \right\|/R<1/50. Therefore, equation (?) implies

Let {\bf{u}}_{i_\star}\in{\mathbb{R}}^T be the i_\starth column of {\bf U}. Recall that {\bf U}^\top{\bf U}={\bf I}_T, we then apply equations (?) and (?) to give

Note that the definition of i_\star indicates y_{i_\star}=\alpha({\bf{x}})\langle{\bf{u}}_{i_\star},{\bf{x}}\rangle and |y_{i_\star}|\le1, which results

For the second term on the right-hand side of inequality (?), we have

where the first inequality is based on Cauchy–Schwarz inequality, the second inequality is based on equation (?), the third inequality is based on equation (Proof) and the definition (?), and the last inequality is based on the condition \left\| {\bf{x}} \right\|<20\sqrt T/\mu and the setting R=1000\sqrt T/\mu. For the last term on the right-hand side of inequality (?), equation (?) and the setting \mu\le10^{-2} implies

Combining equations (?), (?), (?), and (?) with equation (?), we obtain

Since we have \left\| {\bf{u}}_{i_\star} \right\|=1, the Cauchy–Schwarz inequality yields

Combining equation (?) and equation (?), we conclude that whenever there exists some i\in[T] such that |[{\bf U}^\top\rho_R({\bf{x}})]_i|\le1, it holds \left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\|\ge 1/4.

The upper bound for the Hessian. We now establish the gradient-dependent upper bound for the Hessian. For all vectors {\bf{h}},{\bf{k}}\in{\mathbb{R}}^d, the chain rule implies

Taking the supremum on {\bf{h}} and {\bf{k}} over the unit ball and and using the fact \left\| {\bf U} \right\|=1, we obtain

where the second inequality is based on equations (Proof), (Proof), and (?), the second last two inequalities are based on the settings R=1000\sqrt T/\mu and \mu\le10^{-2}.

If there exists some i\in[T] such that |y_i|\le1. According to equation (3.3) from Lemma 3.3, we have \left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\|\ge1/4. Combining this with equation (?) and the setting \mu<1 yields

We then focus on the case that |y_i|>1 holds for all i\in[T]. If \left\| {\bf{x}} \right\|\ge20\sqrt T/\mu, then equations (?) and (?) imply

If \left\| {\bf{x}} \right\|<20\sqrt T/\mu, then equation (?) gives

The condition |y_i|>1 holds for all i\in[T] indicates that we can combine equation (?) and Lemma 3.1(c) to achive

We denote the index j\in[T] such that \phi'(y_j)=\max_{i\in[T]}\phi'(y_i) and analyze the right-hand side of inequality (?) by the following two cases.

1. If |y_j|\ge\log(1/\mu), then Lemma 3.1(c) implies

   \max_{i\in[T]}\phi'(y_i) = \operatorname{sech}^2(y_j) \le4\mathrm e^{-2|y_j|} \le4\mu^2 \le\mu\log\frac1\mu,

   where the first inequality follows from \cosh t\ge\mathrm e^{|t|}/2, and the last inequality uses \mu\le10^{-2}.

2. If |y_j|<\log(1/\mu), then the identity y_j=\alpha({\bf{x}})\langle{\bf{u}}_j,{\bf{x}}\rangle implies

   |\langle{\bf{u}}_j,{\bf{x}}\rangle| < \frac{\log(1/\mu)}{\alpha({\bf{x}})}.

   Following the derivation of equation (?) by replacing {\bf{u}}_{i_\star} with {\bf{u}}_{j}, we obtain

   \begin{aligned} \langle {\bf{u}}_j ,\nabla H_{{\bf U},\mu}({\bf{x}})\rangle &=\frac{\alpha({\bf{x}})\partial\bar f_T({\bf{y}})}{\partial y_{j}}-\frac{(\alpha({\bf{x}}))^3}{R^2}\langle{\bf{u}}_j,{\bf{x}}\rangle\langle{\bf{x}},{\bf U}\nabla\bar f_T({\bf{y}})\rangle +\mu\langle{\bf{u}}_j,{\bf{x}}\rangle \\ &=-\alpha({\bf{x}})\phi'(y_j)-\frac{(\alpha({\bf{x}}))^3}{R^2}\langle{\bf{u}}_j,{\bf{x}}\rangle\langle{\bf{x}},{\bf U}\nabla\bar f_T({\bf{y}})\rangle +\mu\langle{\bf{u}}_j,{\bf{x}}\rangle, \end{aligned}

   where the last step is based on equation (?). Since we have \phi'(y_j)=\max_{i\in[T]}\phi'(y_i) and \left\| {\bf{u}}_j \right\|=1, applying the triangle inequality on above equation implies

   \begin{aligned} \alpha({\bf{x}})\max_{i\in[T]}\phi'(y_i)\le\left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\|+\frac{(\alpha({\bf{x}}))^3}{R^2}|\langle{\bf{u}}_j,{\bf{x}}\rangle| \left|\langle{\bf{x}},{\bf U}\nabla\bar f_T({\bf{y}})\rangle\right| + \mu|\langle{\bf{u}}_j,{\bf{x}}\rangle|. \end{aligned}

   For the second term on the right-hand side of equation (?), applying Cauchy–Schwarz inequality, equation (Proof) with the fact {\bf U}^\top{\bf U}={\bf I}_T imply

   \begin{aligned} |\langle{\bf{x}},{\bf U}\nabla\bar f_T({\bf{y}})\rangle| \le \left\| {\bf{x}} \right\|\left\| \nabla\bar f_T({\bf{y}}) \right\|\le9\sqrt T\left\| {\bf{x}} \right\|. \end{aligned}

   Combining with equation (?), we obtain

   \begin{aligned} \begin{split} & \frac{(\alpha({\bf{x}}))^3}{R^2}|\langle{\bf{u}}_j,{\bf{x}}\rangle|\left|\langle{\bf{x}},{\bf U}\nabla\bar f_T({\bf{y}})\rangle\right| \\ \le & \frac{9(\alpha({\bf{x}}))^2\sqrt T\left\| {\bf{x}} \right\|\log(1/\mu)}{R^2} \\ < & \frac{9\mu\log(1/\mu)}{50000}<\frac{\mu\log(1/\mu)}{5000}, \end{split} \end{aligned}

   where the second inequality follows from \alpha({\bf{x}})\le1, \left\| {\bf{x}} \right\|<20\sqrt T/\mu, and R=1000\sqrt T/\mu. Substituting equations (?) and (?) into equation (?), we obtain

   \begin{aligned} \alpha({\bf{x}})\max_{i\in[T]}\phi'(y_i) \le \left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\| + \left(\frac1{5000}+\frac1{\alpha({\bf{x}})} \right)\mu\log\frac1\mu. \end{aligned}

   Dividing equation (?) by \alpha({\bf{x}})>0 and applying equation (?), we obtain

   \begin{aligned} \begin{split} \max_{i\in[T]}\phi'(y_i) &\le \frac1{\alpha({\bf{x}})} \left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\|+ \left(\frac1{5000\alpha({\bf{x}})} + \frac1{(\alpha({\bf{x}}))^2} \right)\mu\log\frac1\mu \\ &\le 2\left( \left\| \nabla H_{{\bf U},\mu}({\bf{x}}) \right\| +\mu\log\frac1\mu \right). \end{split} \end{aligned}

Combining equations (?) and (?), we conclude that

We now return to the bound of Hessian. The first inequality in equation (?) implies

where the second inequality is based on equations (?), (Proof), (Proof),and (?); the third inequality inequality follows from equation (?) and setting R=1000\sqrt T/\mu; the inequality uses the fact \mu\le10^{-2} which implies \mu\le\mu\log(1/\mu).

Combining equations (?), (?), and (?), we conclude that for every {\bf{x}}\in{\mathbb{R}}^d, it holds

This completes the proof.

Recall that the definition of f_{{\bf U}}({\bf{x}})=aH_{{\bf U},\mu}({\bf{x}}/b) in equation (?) implies

We verify the properties of the function f_{{\bf U}}({\bf{x}}) as follows.

The proof of statement (a). Combining the gradient-dependent curvature bound equation (3.3) and equation (Proof), we have

where the last step follows from the setting b=480/L_1 and a/b=16\epsilon in equation (3.4). For the term \mu\log(1/\mu), recall we set

in equation (3.4), which implies

where the first inequality uses \log t\le t-1 for all t>0. Combining with the definition of \delta_0 and \mu in equation (?), it holds

Combining equation (?) and equation (?), we know that for all {\bf{x}}\in{\mathbb{R}}^d, it holds

Following Corollary A.4 of [zhang2020improved] by replacing their parameters L_0, L_1, and c with L_0/2, L_1/2, and 1, respectively, we obtain

for all {\bf{x}},{\bf{y}}\in{\mathbb{R}}^d with \left\| {\bf{y}}-{\bf{x}} \right\|\le 1/L_1\le 2/L_1. Thus, the function f_{{\bf U}} is (L_0,L_1)-smooth.

The proof of statement (b). Since a,b>0, the definition of the function f_{{\bf U}} in equation (?) and the initial optimality gap bound equation (3.3) imply

where the second equality follows from the fact that the mapping {\bf{x}}\mapsto{\bf{x}}/b is a bijection on {\mathbb{R}}^d.

The proof of statement (c). Let i be the index i\in[T] such that

By the gradient formula in equation (Proof) and equation (3.3), we obtain

where the last equality follows from the setting a/b=16\epsilon in equation (3.4).

The verify properties of functions \bar F_T and Q_T.

The proof of statement (a). According to equation (?) and the product rule, we know that

We denote

For any m\in\{0,1,\ldots,T\} and i\ge m+2, we have |y_{i-1}|\le1/2. Hence, Lemma 3.1(a) and equation (Proof) implies

Combing above results with equation (?) gives A_i({\bf{y}})=0 and \nabla A_i({\bf{y}})={\bf{0}}, which implies

In the case of m=0, combing equations (Proof) and (?) implies

In the case of m\in[T], we have |y_m|>1/2. Hence, equation (Proof) and equation (Proof) imply

Note that for all i\le m, equations (?) and (Proof) yield

respectively. This implies

Combining equations (?), (?) and (?), we obtain

We now establish the upper bound on \left\| \nabla Q_T({\bf{y}}) \right\| by considering the following three cases.

1. In the case of m=0, we have A_1({\bf{y}})=-\phi(y_1), so that Lemma 3.1(c) implies |A_1({\bf{y}})|<2 and \left\| \nabla A_1({\bf{y}}) \right\|\le1. By equation (?), equation (Proof), and Lemma B.2, we obtain

   \begin{aligned} \left\| \nabla Q_T({\bf{y}}) \right\| &\le |\Theta_1({\bf{y}})|\left\| \nabla A_1({\bf{y}}) \right\|+|A_1({\bf{y}})|\left\| \nabla\Theta_1({\bf{y}}) \right\| \\ &\le1+2\cdot120 =241<249. \end{aligned}

2. In the case of m\in[T-1], combining the definition of A_i({\bf{y}}) in equation (?) with equation (A.1) and Lemma 3.1(c) imply

   |A_{m+1}({\bf{y}})| = |\psi(-y_{m})\phi(-y_{m+1})-\psi(y_{m})\phi(y_{m+1})| \le 2(\psi(-y_m)+\psi(y_m)) \le 2.

   Moreover, combining the definition of A_i({\bf{y}}) in equation (?) with equation (A.1), equation (A.1), and Lemma 3.1(c), we have

   \begin{aligned} &\left|\frac{\partial A_{m+1}({\bf{y}})}{\partial y_m}\right| = |\psi'(-y_{m})\phi(-y_{m+1})+\psi'(y_{m})\phi(y_{m+1})| \le 2|\psi'(-y_m)+\psi'(y_m)| \le8,\\ &\left|\frac{\partial A_{m+1}({\bf{y}})}{\partial y_{m+1}}\right| = |\psi(-y_{m})\phi'(-y_{m+1})+\psi(y_{m})\phi'(y_{m+1})| \leq |\psi(-y_m)+\psi(y_m)| \le1, \end{aligned}

   Since the function A_{m+1}({\bf{y}}) depends only on y_m and y_{m+1}, the above results leads to

   \left\| \nabla A_{m+1}({\bf{y}}) \right\| \le\sqrt{8^2+1^2} =\sqrt{65}<9.

   Therefore, combining equation (?) with equations (?), (Proof), (?), and (?), and Lemma B.2, we have

   \begin{aligned} \begin{split} \left\| \nabla Q_T({\bf{y}}) \right\| &\le |\Theta_{m+1}({\bf{y}})| \left\| \nabla A_{m+1}({\bf{y}}) \right\| +|A_{m+1}({\bf{y}})| \left\| \nabla\Theta_{m+1}({\bf{y}}) \right\| \\ &\le9+2\cdot120 =249. \end{split} \end{aligned}

3. In the case of m=T, equation (?) directly gives \left\| \nabla Q_T({\bf{y}}) \right\|=0\le249.

Therefore, the above derivations conclude \left\| \nabla Q_T({\bf{y}}) \right\|\le249 holds for all {\bf{y}}\in{\mathbb{R}}^T.

The proof of statement (b). Differentiating equation (?) obtains

The distribution Z\sim\operatorname{Bernoulli}(q) indicates {\mathbb{E}}_Z[Z/q-1]=0, resulting

We then verify the property of probability-q zero-chain for {\bf{g}}_T({\bf{y}};Z). Fix given {\bf{y}}\in{\mathbb{R}}^T, we let

We show that {\bf{g}}_T({\bf{y}};Z) is q-probability zero-chain by verifying conditions (4.1) and (4.1) as follows.

1. If s<T, the definition of s implies |y_{s+1}|,\ldots,|y_T|\le1/4. Hence, when Z=0, we can apply Lemma B.3(a) to achieve

   \begin{aligned} \operatorname{prog}_0({\bf{g}}_T({\bf{y}};0)) \le s \quad\text{and}\quad {\bf{g}}_T({\bf{y}};0) = {\bf{g}}_T(\Pi_s({\bf{y}});0) \quad \text{for all } {\bf{y}}\in{\mathbb{R}}^T. \end{aligned}

   Recall that \mathbb P(Z=0)=1-q, then equation (?) verifies that condition equation (4.1) with \theta=q. When Z=1, wen can apply Lemma B.3(b) to achieve

   \begin{aligned} \operatorname{prog}_0({\bf{g}}_T({\bf{y}};1)) \le s+1 \quad\text{and}\quad {\bf{g}}_T({\bf{y}};1) = {\bf{g}}_T(\Pi_{s+1}({\bf{y}});1) \quad \text{for all } {\bf{y}}\in{\mathbb{R}}^T. \end{aligned}

   According to Lemma B.3(a), we know that equation (?) also holds when Z=0. Hence, we have verifies condition (4.1) with \theta=q.

2. If s=T, we have \operatorname{prog}_{1/4}({\bf{y}})=T and \Pi_{1+\operatorname{prog}_{1/4}}({\bf{y}})=\Pi_{1+\operatorname{prog}_{1/4}}({\bf{y}})={\bf{y}}. Therefore, for all {\bf{y}}\in{\mathbb{R}}^T, all conditions

   \begin{aligned} \operatorname{prog}_0({\bf{g}}_T({\bf{y}};Z)) \le \operatorname{prog}_{1/4}({\bf{y}}),\quad {\bf{g}}_T({\bf{y}};Z) = {\bf{g}}_T\bigl( \Pi_{\operatorname{prog}_{1/4}({\bf{y}})}({\bf{y}});Z \bigr), \\ \operatorname{prog}_0({\bf{g}}_T({\bf{y}};Z)) \le1+\operatorname{prog}_{1/4}({\bf{y}}),\quad {\bf{g}}_T({\bf{y}};Z) = {\bf{g}}_T\bigl( \Pi_{1+\operatorname{prog}_{1/4}({\bf{y}})}({\bf{y}});Z \bigr) \end{aligned}

   holds in both cases of Z=0 and Z=1. This verifies equations (4.1) and (4.1).

Hence, we conclude that {\bf{g}}_T is a probability-q zero-chain.

We let {\bf{y}}:={\bf U}^\top\rho_R({\bf{x}}/b). By the chain rule, the stochastic oracle defined in equation (?) has the form of

Combining the above result with equations (?), (?), (Proof), and (?), we obtain

Unbiasedness. Since Z\sim\operatorname{Bernoulli}(q) with q\in(0,1], taking the expectation on equation (Proof) achieves

Bound on the pth central moment. Combing equations (Proof), (Proof), and (?), we have

where we use the setting \left\| {\bf U} \right\|=1 and a/b=16\epsilon from equation (3.4). Since Z\sim\operatorname{Bernoulli}(q) and p\in(1,2], we have

where the inequalities follow from 0<q\le1 and p>1. Consequently, combining equations (?) and (?) imply

If q<1, then \sigma>0 and the definition of q gives q^{p-1}=2(3984)^p\epsilon^p\sigma^{-p}. Substituting this identity into equation (?), we obtain

If q=1, then Z=1 almost surely. Hence, equation (?) implies

Combining equation (?) and equation (?), we conclude that for every {\bf{x}}\in{\mathbb{R}}^d,

According to Lemma 4.2, we know that the gradient evaluation {\bf{g}}_T({\bf{y}};Z) is a probability-q zero-chain. Moreover, the definition of \rho implies \left\| \rho_R({\bf{x}}/b) \right\|\le R for every {\bf{x}}\in{\mathbb{R}}^d. Note that the proof of Lemma 5 of [arjevani2023lower] only require the oracle is a probability zero-chain, so that we can use this lemma with K=1 to achieve

for all t\le(T-\log(2/\delta))/(2q). Hence, we have

According to Lemma 3.3, we obtain

Based on the definition

with a/b=16\epsilon, it follows that

This completes the proof.

We follow the setting of a, b, \mu, and R by equations (3.4) and (?). For \epsilon\le\hat c L_1\Delta with \hat c=1/122880, we let

Fix p\in(1,2]. According to Lemma 3.4(a) and (b), the function f_{{\bf U}} is (L_0,L_1)-smooth and satisfies

where the last inequality follows from equation (Proof). Thus, we conclude f_{{\bf U}}\in\mathcal F(L_0,L_1,\Delta).

Following the setting of q in equation (4.3), Lemma 4.3 implies the oracle \mathsf O_{f_{{\bf U}}}({\bf{x}};Z) satisfies is unbiased and has pth bounded central moment.

We let {\bf U}\sim\operatorname{Unif}(\operatorname{St}(d,T)) and use P_f to denote the induced distribution of f_{{\bf U}}. Taking sufficient large d such that

which satisfies equation (?) with \delta=1/2. Applying Lemma 4.4, we know that for all randomized algorithm \mathsf A\in\mathcal A_{\mathrm{rand}} and t\le N with

we have

This implies

where the expectation and probability are taken over {\bf U}, the oracle randomness, and the internal randomness of \mathsf A. Since the distribution P_f is supported on \mathcal F(L_0,L_1,\Delta) and the oracle \mathsf O_{f_{{\bf U}}} condition equation (4.3), the distributional complexity defined in equation (?) implies

We now provide the lower bound for N. According to equation (?), we have

where the first inequality uses \lfloor x\rfloor\ge x-1, and the last two inequalities follow from q\le1, \log4<2, and T\ge8. Moreover, by equation (Proof) and L_1\Delta/(15360\epsilon)\ge8, we obtain

Additionally, the definition of q implies

where the inequality follows from \max\{u,v\}\ge(u+v)/2 for u,v\ge0 and 2^{-\frac1{p-1}}(3984)^{-\frac p{p-1}}\le1. For any given p\in(1,2], combining equations (?), (?), (?), and (?) achieves

This finishes the proof.

For any given p\in(1,2], recall that [liu2025nonconvex, Theorem 3.3] have provided a lower bound for standard smooth optimization under the p-BCM condition, i.e., there exists \tilde c>0 such that for all L_0,\Delta,\sigma>0, and \epsilon\le\tilde c\sqrt{L_0\Delta}, it holds

Since \mathcal F(L_0,0,\Delta)\subset\mathcal F(L_0,L_1,\Delta), the above lower bound implies

Combining the result of Theorem 4.5, we know that for any given p\in(1,2], there exists numerical constant c=\min\{\hat c, \tilde c\} such that for all L_0,L_1,\Delta,\sigma>0, and \epsilon\le\hat c\min\{\sqrt{L_0\Delta},L_1\Delta\}, it holds

This finishes the proof.