General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization

Schedule-free SGD (?), with an appropriate choice of parameters \gamma, c_t, \kappa_t, achieves optimal rates for nonsmooth and nonconvex F.

If F is L-smooth, then an (L^2 \varepsilon^{-1},\varepsilon)-stationary point {\mathbf x} of F satisfies \left\|\nabla F({\mathbf x})\right\|\leq 2\varepsilon. Moreover, if F is H-second-order-smooth, then an (H/2,\varepsilon)-stationary point {\mathbf x} of F satisfies \left\|\nabla F({\mathbf x})\right\|\leq 2\varepsilon.

Consider the iterates generated according to ?. For \beta \in (0, 1) and D > 0, define the comparators for online learner as follows:

Then, as long as the regularization strength satisfies \mu \geq 8\lambda D \left(1 + { C_{{\mathbf x}}}{(1-\beta)^{-2}} \right), the following holds:

Let \beta \in (0,1), \mu \geq 0, \eta > 0, and a sequence of vectors \{\mathbf{g}_t\}_{t=1}^T. Suppose that \mathbb{E}\left\|\mathbf{g}_t\right\|^2 \leq G^2 + \sigma^2 for all t\in[T]. Consider an online learner initialized with {\boldsymbol \delta}_1 = \bm{0} and updated as follows:

Then, this online learner with \eta = \frac{2}{G + \sigma} \left\|\mathbf{u}\right\| \sqrt{1-\beta} achieves the following discounted regret bound:

Consider the iterates generated according to ?, where the online learner \mathcal{A} achieves the discounted regret given by (2). Let \varepsilon > 0 be such that \varepsilon \leq \frac{7}{2}(G + \sigma) and let T \geq 49 (G + \sigma)^2 \varepsilon^{-2}. Then, there exists a choice of parameters \beta=\beta_\star, D=D_\star, \mu=\mu_\star such that with \overline{\mathbf{y}}_t and \tau as specified by Definition 3 and Definition 4, the following holds:

The choice of parameters is summarized as follows:

• The discounting factor is \beta_\star = 1 - (\frac{\varepsilon}{7(G + \sigma)})^2.

• The comparator norm is D_\star = \frac{1}{4} \lambda^{-1/2} \varepsilon^{1/2} \left(1+\frac{49(G + \sigma)^2 }{\varepsilon^2} \sqrt{C_{{\mathbf x}}} \right)^{-1}.

• The regularization strength is \mu_\star = 2 \lambda^{1/2}\varepsilon^{1/2} \left(1 + \frac{49(G + \sigma)^2 }{\varepsilon^2}\sqrt{C_{{\mathbf x}}} \right).

Consider the iterates of ?. Under the setting of Theorem 2, it holds that \mathbb{E}_{\tau} \left\| \nabla F(\overline{\mathbf{y}}_{\tau})\right\|^{[\lambda]} \leq 4\varepsilon, provided that

Consider the iterates of ?. Under the setting of Theorem 2, the following holds:

provided that T \geq 49 (G + \sigma)^2 \varepsilon^{-2}.

Consider the iterates of ?. Under the setting of Theorem 2, it holds that

provided that N \geq 4\Delta_F \lambda^{1/2} \varepsilon^{-3/2} and T \geq 49 (G + \sigma)^2 \varepsilon^{-2}.

Consider the iterates of ? with the parameter choices as in Theorem 2, i.e., \beta = \beta_\star, D = D_\star, \mu = \mu_\star, and the parameters of ? chosen as:

Then, it holds that \mathbb{E}_{\tau} \left\| \nabla F(\overline{\mathbf{y}}_{\tau})\right\|^{[\lambda]} \leq 5\varepsilon, provided that

Under ? and using the extrapolated iterates defined in (?), the following holds:

With the parameter choices given in Corollary 6, we have \zeta = \zeta_\star = 1 - \Theta \left(\frac{\varepsilon^2}{(G + \sigma)^2} \right). Specifically, the parameters in schedule-free SGD are set as follows:

• \kappa_t is chosen uniformly from [\zeta_\star, 1], implying that 1 - \Theta \left(\frac{\varepsilon^2}{(G + \sigma)^2}\right) \leq \kappa_t \leq 1 for all t. This selection ensures that \kappa_t remains close to 1.

• The step size for the base SGD ({\mathbf z}-trajectory) is set to \frac{\eta_\star}{1-\zeta_\star}, making it \Theta\left(\frac{(G + \sigma)^2}{\varepsilon^2}\right) times larger than the OMD step size, \eta_\star.

For \beta\in(0,1), consider the iterates generated as per ?.

Consider the following definitions:

• For each t \in [T], let \mathbf{u}_t \coloneqq -D \frac{\sum_{s=1}^t\beta^{t-s}\nabla F(\mathbf{y}_s)}{\left\|\sum_{s=1}^t\beta^{t-s}\nabla F(\mathbf{y}_s)\right\|}.

• For each t \in [T], let \mathbf{Y}_t be the random iterate distributed over \{\mathbf{y}_s\}_{s=1}^t as:

  \begin{aligned} \Pr(\mathbf{Y}_t= \mathbf{y}_s) = q_{t,s} \coloneqq \beta^{t-s} \cdot \frac{1-\beta}{1-\beta^t} \quad \text{for $s=1,2,\dots, t$.} \end{aligned}

Then, the following upper bound holds:

For \beta\in(0,1), consider the iterates generated as per ?

For \beta\in(0,1), consider the iterates generated as per ?.

Consider the following definitions:

• For each t \in [T], let \mathbf{u}_t \coloneqq -D \frac{\sum_{s=1}^t\beta^{t-s}\nabla F(\mathbf{y}_s)}{\left\|\sum_{s=1}^t\beta^{t-s}\nabla F(\mathbf{y}_s)\right\|}.

• For each t \in [T], let \mathbf{Y}_t be the random iterate distributed over \{\mathbf{y}_s\}_{s=1}^t as:

  \begin{aligned} \Pr(\mathbf{Y}_t= \mathbf{y}_s) = q_{t,s} \coloneqq \beta^{t-s} \cdot \frac{1-\beta}{1-\beta^t} \quad \text{for $s=1,2,\dots, t$.} \end{aligned}

Then, for any n\in [T], we have

For each n \in [T] the following holds:

The following holds:

For each n \in [T], it holds that

For 1\leq t\leq n\leq T, it holds that