Parameter-Free Adaptive Extragradient Methods for Generalized-Smooth Variational Inequalities

By the first term in the stepsize rule (?), \lambda_{k+1}\le(1+\eta_k)\lambda_k. Iterating this inequality and using \log(1+x)\le x for x\ge0, we obtain

which proves \mathbf{(i)}.

We next prove \mathbf{(iii)}. The stepsize rule (?) gives \lambda_{k+1}-\lambda_k\le\eta_k\lambda_k\le\bar\lambda\eta_k. Hence

Since \sum_{k=0}^{\infty}\eta_k<\infty, it follows that

Moreover, for every N\ge0,

Since \lambda_{N+1}\ge0,

Letting N\to\infty proves the second assertion in (?), and therefore (?) follows.

Finally, (?) implies that \{\lambda_k\} is a Cauchy sequence in \mathbb{R}. Thus it converges to a finite limit \lambda_\infty\ge0, proving \mathbf{(ii)}.

Since

the projection inequality (2.2) gives, for every y\in U,

Using

we can rewrite (Proof) as

Because t_k=P_U\bigl(z_k-\lambda_kF(z_k)\bigr), the projection optimality condition (2.2), with z_{k+1}\in U, implies

In addition, the Cauchy–Schwarz and Young inequalities yield

Substituting these two estimates into (Proof) gives (3.3).

(i) By (?) and the nonexpansiveness of P_U, for every k\ge1,

Suppose first that \alpha\in(0,1). Applying Proposition 2.1 with y=t_k and y'=z_k gives

Moreover, (Proof) implies

Using (?), (Proof), and (Proof), we obtain

Similarly, when \alpha=1,

Hence, in both cases,

Since

Lemma 3.3, (?), and (Proof) imply that, for every y\in U,

Taking y=t^*\in U^* and using p-quasi-sharpness proves (?).

(ii) Fix any t^*\in U^*. Summing (?) from k=1 to n and dropping the nonpositive terms gives

By Lemma 3.1, the series on the right-hand side is convergent. Hence \{z_k\} is bounded. Since U is closed and F is continuous, there exists C>0 such that

If \alpha\in(0,1), then (Proof), (Proof), and (?) give

Similarly, if \alpha=1,

Therefore, whenever F(z_k)\ne F(t_k),

If F(z_k)=F(t_k), the corresponding candidate is interpreted as +\infty. By the definition of \underline{\lambda}, \lambda_0\ge\underline{\lambda} and \lambda_1\ge\underline{\lambda}. Suppose that \lambda_k\ge\underline{\lambda} for some k\ge1. The first term in (?) satisfies (1+\eta_k)\lambda_k\ge\underline{\lambda}. The preceding estimates show that the second candidate in (?) is also no smaller than \underline{\lambda}. Finally, by (?), the third candidate satisfies 1/\|F(z_{k+1})\|^2\ge1/C^2\ge\underline{\lambda}, with the convention 1/0=+\infty. Hence \lambda_{k+1}\ge\underline{\lambda}. By induction, (?) holds for all k\ge0.

(iii) It follows from (?) and Lemma 3.3 that, for every y\in U,

By Lemma 3.1 and (?), \{\lambda_k\} converges to a positive limit. Hence \frac{\lambda_k}{\lambda_{k+1}}\to1.

Therefore, there exists k_0\ge1 such that

Fix an arbitrary t^*\in U^*. Taking y=t^*\in U^* in (?) and applying p-quasi-sharpness and (?) give

Applying Lemma 2.3 to the tail sequence in (?) shows that \{\|z_k-t^*\|^2\} converges and

Since only finitely many indices precede k_0, we get

Consequently,

We next show that every accumulation point of \{z_k\} belongs to U^*. By part \mathbf{(ii)}, the sequence \{z_k\} is bounded. Hence it has an accumulation point in \mathbb{R}^m. Let \{z_{k_i}\} be a convergent subsequence such that z_{k_i}\to\bar t. Since z_{k_i}\in U and U is closed, we have \bar t\in U. Moreover, (?) gives \|t_{k_i}-z_{k_i}\|\to0, and therefore t_{k_i}\to\bar t. Since the distance function is continuous, (?) yields

Since U^* is closed, \bar t\in U^*. Thus every accumulation point of \{z_k\} belongs to U^*.

Finally, (?) implies that the tail sequence \{z_k\}_{k\ge k_0} is Fejér monotone with respect to U^*. Applying Lemma 2.4 to this tail sequence, we conclude that there exists \bar t\in U^* such that z_k\to\bar t. Together with \|z_k-t_k\|\to0, this further gives t_k\to\bar t. Hence both \{z_k\} and \{t_k\} converge to the same solution \bar t\in U^*.

By Theorem 3.4(iii), there exists \bar t\in U^* such that t_k\to\bar t. Hence \operatorname{dist}(t_k,U^*)\to0.

Therefore, there exists B>0 such that

Let k_0\ge1 be such that (?) holds for all k\ge k_0. For each such k, let z_k^* be the projection of z_k onto U^*, i.e.,

Taking t^*=z_k^* in (?) gives

Moreover, the triangle inequality yields

(i) Suppose 0<p\le2. Squaring (?) and using (Proof), we obtain

By the definition of \rho in (?),

Hence

Combining this inequality with (?) yields

which implies

Then (?) holds for every k\ge0 by setting

(ii) Suppose p>2. By (?), the fact that {\left(a + b\right)}^{p} \leq {2}^{p - 1}\left(a^{p} + b^{p}\right), and \|z_k-t_k\|\le\sqrt{\bar{\lambda}}, we have

Thus

Combining this estimate with (?) gives

Applying Lemma 2.6 to (?) with q=(p-2)/2 yields

where

If \operatorname{dist}(z_{k_0},U^*)=0, then (?) implies \operatorname{dist}(z_k,U^*)=0 for all k\ge k_0, and the result follows immediately. Otherwise, (?) implies, for k\ge k_0+1,

For k\ge k_0+1, k-k_0\ge\frac{k+1}{k_0+2}, and hence

Thus (?) holds for every k\ge0 with

The first candidate in (?) gives \lambda_{k+1}\le(1+\eta_k)\lambda_k pathwise. Hence the proof of (4.1) is identical to that of Lemma 3.1. Also,

Since \sum_k\eta_k<\infty, the positive variations (\lambda_{k+1}-\lambda_k)_+ are summable pathwise. The identity

and \lambda_{N+1}\ge0 imply summability of the negative variations. This proves (4.1), and therefore \{\lambda_k\} is Cauchy and convergent. The expectation bounds in (4.1) follow from the same deterministic majorants and Tonelli’s theorem.

Fix any t^*\in U^* and k\ge1. Since t^{k+1}=P_U\left(\bar t^k-\lambda_k\widehat F_k\right), the inequality (2.2) gives

Splitting t^*-t^{k+1}=t^*-t^{k+\frac12}+t^{k+\frac12}-t^{k+1}, we obtain

Applying (2.2) with u=t^{k+1}\in U and v=\bar t^k-\lambda_kF(s^k) yields

Equivalently,

Using \widehat F_k=F(s^k)+F_{\xi_k}(t^{k+\frac12})-F_{\xi_k}(s^k), it follows that

Substituting this estimate into (Proof) and using the identity

we obtain

By Young’s inequality,

Therefore,

Since \bar t^k=\beta t^k+(1-\beta)s^k, the standard convex-combination identity gives, for every u\in\mathbb R^m,

Applying this identity with u=t^* and u=t^{k+\frac12}, and subtracting the resulting equalities, yields

Substituting this equality into (Proof) gives

We next control the last term. By the second candidate in (?),

Using \lambda_k^2\le\lambda_{k+1}^2+2\lambda_k(\lambda_k-\lambda_{k+1})_+, we have

Since s^k\in U, nonexpansiveness of P_U, (?), and (?) yield

If \alpha\in(0,1), by Assumption 4, Proposition 2.1, and (?),

If \alpha=1, Proposition 2.1 similarly gives

where we used \sqrt{\lambda_k}\|F_{\xi_k}(s^k)\| \le1. Thus, in both cases, by the definition of \bar M,

Combining (?) and (?) gives

Substituting this estimate into (Proof) yields

Finally, taking conditional expectation with respect to \mathcal F_k, using (?), gives

where the last inequality is due to p-quasi-sharpness. Substituting this inequality into the preceding estimate proves (4.2).

For every k\ge1, by (2.2), (?) and (?), we have

where the last inequality follows from the second inequality by adding and subtracting the term \lambda_{k+1}\|F_{\xi_k}(t^{k+\frac{1}{2}})-F_{\xi_k}(s^k)\|. Using (?), we obtain

which together with (?) and (?) gives

Condition (4.3) is equivalent to w>(1+\beta)R. Hence, on A_k,

when \bar M>0. Therefore,

Lemma 4.1 implies

If \bar M=0, then (?) and (4.3) imply directly that A_k does not occur for any k\ge1. In either case, the possible event at the single initial index k=0 does not affect the conclusion.

\mathbf{(i)} Conditioning on \mathcal G_k, both t^{k+1} and A_k are measurable. If A_k occurs, then s^{k+1}=t^{k+1}. If A_k^c occurs, then s^{k+1}=t^{k+1} with probability P_1 and s^{k+1}=s^k with probability 1-P_1. Hence, for k\ge1,

Rearranging (Proof) gives

We next bound the last term in (Proof). By the difference-of-squares identity and Young’s inequality,

Moreover, (?) and (\lambda_k-\lambda_{k+1})_+\le\lambda_k\le\bar\lambda imply

Since

it follows from (?) and (?) that

By the definition of \Phi_{k+1}(t^*), (Proof), and (?),

Taking conditional expectation with respect to \mathcal F_k and using the tower property, together with the \mathcal F_k-measurability of \Phi_k(t^*), yields

Applying Lemma 4.2 to the first term on the right-hand side of (?), and using

we obtain

It remains to verify the summability of the error terms in (?). By the proof of Lemma 4.3, (?) holds. Taking conditional expectation in (?) gives

By Tonelli’s theorem, the tower property, and Lemma 4.1, we get

Therefore,

Together with (?), this further gives

Hence both the multiplicative error and the additive error in (?) are summable almost surely.

Since \beta\in\left(0,\frac12\right),

all the descent terms in (?) are nonnegative. Applying Lemma 2.5 to the tail sequence \{\Phi_k(t^*)\}_{k\ge1}, we conclude that

Moreover,

almost surely.

Finally, convergence of \Phi_k(t^*) implies that \{\Phi_k(t^*)\} is bounded almost surely. Since \beta>0 and (1-\beta)/P_1>0, the definition of \Phi_k(t^*) then implies that both \{t^k\} and \{s^k\} are bounded almost surely. This proves part \mathbf{(i)}.

(ii) We next establish a positive lower bound for the stepsizes. By part \mathbf{(i)}, \{s^k\} is bounded almost surely. Since U is closed, N<\infty, and every F_i is continuous, it follows that

Suppose first that \alpha\in(0,1). By Assumption 4, Proposition 2.1, (?), and (?),

Hence, whenever F_{\xi_k}(t^{k+\frac12})\ne F_{\xi_k}(s^k), the second candidate in (?) satisfies

If the denominator is zero, the corresponding candidate is interpreted as +\infty, so the same lower bound remains valid.

If \alpha=1, Proposition 2.1, (?), and (?) give

Therefore, whenever the denominator is nonzero,

By (?),

Moreover, \lambda_0\ge\underline{\lambda} and \lambda_1\ge\underline{\lambda}. Suppose that \lambda_k\ge\underline{\lambda} for some k\ge1. Then the first candidate in (?) satisfies (1+\eta_k)\lambda_k\ge\lambda_k\ge\underline{\lambda}. By (?) or (?), the second candidate is also no smaller than \underline{\lambda}. Finally,

Thus every candidate in (?) is bounded below by \underline{\lambda}, and hence \lambda_{k+1}\ge\underline{\lambda}.

By induction, (?) holds almost surely. This proves part \mathbf{(ii)}.

(iii) We finally prove the convergence of the iterates. By (?) and (?),

almost surely. Hence

almost surely, where the last two limits follow directly from (?). Consequently,

We next show that every accumulation point of \{t^k\} belongs to U^*. Fix a sample path on which the preceding properties hold. By part \mathbf{(i)}, \{t^k\} is bounded. Let \{t^{k_j}\} be a convergent subsequence such that t^{k_j}\to u. By (?), \|t^{k_j+\frac12}-t^{k_j}\|\to0, and hence t^{k_j+\frac12}\to u. Since the distance function is continuous, (?) yields

Since U^* is closed, u\in U^*. Thus every accumulation point of \{t^k\} belongs to U^*.

It remains to show that the accumulation point is unique. Since U^*\subseteq\mathbb R^m is separable, let \{u_j^*\}_{j\ge1} be a countable dense subset of U^*. By part \mathbf{(i)}, for every j\ge1, \Phi_k(u_j^*) converges almost surely. Hence there exists an event of probability one on which

Fix a sample path in this event, and let u',u''\in U^* be two accumulation points of \{t^k\}. Then there exist subsequences such that

By (?), we have

For every j\ge1, the convergence of \Phi_k(u_j^*) implies that its limits along these two subsequences coincide. Therefore,

By density, there exists a subsequence \{u_{j_\ell}^*\} such that u_{j_\ell}^*\to u'. Passing to the limit in (?) along this subsequence gives

Since \beta+\frac{1-\beta}{P_1}>0, we obtain u''=u'. Thus \{t^k\} has a unique accumulation point. Since it is bounded, there exists \bar t\in U^* such that

Finally, (?) implies

This proves (?) and part \mathbf{(iii)}.

By Lemma 4.3, the event A_k occurs only finitely many times almost surely. Hence there exists an almost surely finite random index K_1 such that

almost surely. By Lemma 4.1 and Theorem 4.4(ii), the stepsize sequence converges almost surely: \lambda_k\to\lambda_\infty\ge \underline{\lambda}>0. It follows that \frac{\lambda_k}{\lambda_{k+1}}\to1 almost surely. Consequently,

Since \beta\in(0,\frac12), we have 1-2\beta>0. Thus there exists an almost surely finite random index K_2 such that

almost surely. Taking K=\max\{K_1,K_2\} gives (4.5).

It remains to prove (4.5). By Theorem 4.4(iii), there exists \bar t\in U^* such that s^k\to\bar t almost surely. Hence

For every k\ge1, (?) gives

Therefore,

Since t^{1/2} is finite, (4.5) follows.

By (?), Theorem 4.4(ii), and Lemma 4.5, for every k\ge k_{\rm r}, almost surely,

Fix k\ge k_{\rm r}. The second inequality in (Proof) implies that the snapshot is updated according to the Bernoulli rule. Hence, for every u^*\in U^*,

By the second candidate in (?),

If the operator difference vanishes, this inequality holds trivially. Substituting (Proof) into (Proof) gives

Using (Proof) in (Proof) and dropping the nonpositive term involving \|t^{k+1}-t^{k+\frac12}\|^2, we obtain

Taking conditional expectation with respect to \mathcal F_k, using (?), p-quasi-sharpness, and \lambda_k\ge\lambda_{\min}, gives

Taking conditional expectation with respect to \mathcal F_k in (Proof) and using the tower property, we have

Combining this identity with (?) yields

Since the preceding inequality holds for every u^*\in U^*, and

we obtain

(i) Suppose 0<p\le2. For each k\ge k_{\rm r}, let u_k^*\in U^* satisfy \|t^{k+\frac12}-u_k^*\|=\operatorname{dist}(t^{k+\frac12},U^*). By the definition of \Phi_k and \|a+b\|^2\le2\|a\|^2+2\|b\|^2,

If 0<p<2, (?) implies

For p=2, the same relation holds with B^{2-p}=1. Combining the definition of \rho in (?) and (?) gives

which together with (?) yields

Taking total expectations and iterating gives

Hence (?) holds for every k\ge0 by taking

(ii) Suppose p>2.

Using the same argument as in (?), we obtain

which together with (?) gives

where

Let u_k^*\in U^* be a nearest point to t^{k+\frac12}. By the triangle inequality,

Using (a+b)^p\le2^{p-1}(a^p+b^p) and (?), we obtain

which implies

where

Combining this estimate with (?) gives

Taking total expectations in (?) and applying Jensen’s inequality, since p/2>1, yields

Applying Lemma 2.6 with q=(p-2)/2 gives

If \mathbb E\left[\displaystyle\inf_{u^*\in U^*}\Phi_{k_{\rm r}}(u^*)\right]=0, the result follows immediately. Otherwise, for k\ge k_{\rm r}+1,

Since k-k_{\rm r}\ge\frac{k+1}{k_{\rm r}+2} for k\ge k_{\rm r}+1, we obtain

Thus (?) holds for every k\ge0 by choosing