Existing large-dimensional theory for spectral algorithms resolves either the optimally tuned point or the interpolation limit, but leaves the under-regularized regime unexplored. We study the learning curve and benign overfitting of spectral algorithms in the large-dimensional setting where the sample size and dimension are of comparable order, i.e., n \asymp d^{\gamma} for some \gamma>0. We first consider inner-product kernels on the sphere \mathbb{S}^{d-1} and establish a sharp asymptotic characterization of the excess risk across the full regularization path under various source conditions s \geq 0, where s measures the relative smoothness of the regression function. Our results reveal that the learning curve is not simply U-shaped but instead consists of three distinct regimes: over-regularized, under-regularized, and interpolation regimes. This characterization allows us to fully capture the benign overfitting phenomenon, demonstrating that benign overfitting arises consistently across both the under-regularized and interpolation regimes whenever s is positive but no larger than a critical threshold. We further show that, in the sufficiently regularized regime, the kernel learning curve is recovered by an associated sequence model. Finally, we extend the learning-curve analysis to large-dimensional KRR for a class of kernels on general domains in \mathbb{R}^d whose low-degree eigenspaces satisfy spectral-scaling and hyper-contractivity conditions.
There exist positive constants c_1, c_2, and \gamma such that the sample size satisfies
and we denote \ell_{\gamma} := \lfloor \gamma\rfloor.
\Phi(t) \in \mathcal{C}^{\infty} \left([-1,1]\right) is a fixed function (independent of d) and there exists a non-negative sequence of absolute constants \{a_i\}_{i = -1}^{\infty} such that
where the constant a_{-1} denotes an upper bound on the sum of coefficients. Furthermore, we require a_{i} > 0 for any 0 \leq i \leq \ell_{\gamma} + 2, where \ell_{\gamma} is defined in Assumption 1.
Suppose Assumptions 1 and 2 hold for an absolute constant \gamma>0. Then,
For any integer k = 0, 1, \ldots, \ell_{\gamma} +2, we have
For ease of notation, denote N_{-1}=0 and N_{k} = \sum_{k^{\prime}=0}^{k}N(d, k^{\prime}) = \sum_{k^{\prime}=0}^{k}\Theta_d(d^{k^{\prime}}) = \Theta_d(d^k).
There exists a constant \mathfrak{C} depending only on \gamma, a_0, \ldots, a_{\ell_{\gamma} + 2}, such that for any d \geq \mathfrak{C}, we have for any 0 \leq k \leq \ell_{\gamma}+1:
There exist absolute constants s \geq 0 and R>0, such that we have
Let \left\{ \varphi_\lambda : (0, K_{\max}] \to \mathbb{R}_{\geq 0} \mid \lambda \in [0, 1] \right\} be a family of analytic functions indexed by the regularization parameter \lambda. We call \left\{ \varphi_\lambda \mid \lambda \in [0, 1] \right\} an analytic filter family with qualification \tau \in [1,\infty], if there exist positive absolute constants C_1, C_2, C_3, and C_4, such that:
For all \lambda\in[0,1] and z\in(0,K_{\max}], we have z \varphi_\lambda(z) \in [0, 1] and the function \psi_\lambda(z):= 1- z\varphi_\lambda(z) is non-increasing in z.
For any z\in(0,K_{\max}] and any 1\leq t\leq \tau, there exists a constant \mathfrak{C}_3 only depending on C_3 and t, such that
If \tau<\infty, then there exists a positive constant \mathfrak{C}_{4} only depending on \tau and \lambda_1, such that we have
where \lambda_1 is the largest eigenvalue of \mathscr{K} defined in (?).
When s>0, we also assume that the analytic filter function \varphi_\lambda satisfies the remaining conditions listed in Appendix C of [lu2024saturation] (i.e., equations (30), (31), (C1), and (C2) in [lu2024saturation]). For convenience, these requirements are restated in Appendix J.1.
The conditions restated in Appendix J.1 essentially require that \varphi_\lambda admits an extension to a larger domain, while preserving all relevant properties that hold on the original domain. This allows us to apply the “analytic functional argument” developed by [li2024generalization].
There exist constants c_3,c_4>0 and a constant exponent u \in (0, \infty], such that we have
where the case u = \infty corresponds to \lambda=0 (i.e., the interpolation limit). Define \ell_{\lambda} = \lfloor u \rfloor and \tilde{\ell}=\min\{\ell_{\gamma}, \ell_{\lambda}\}.
When 0 < u < \gamma <1, we require that \psi_\lambda defined in Definition 2.2 satisfies
where K_{0} = \mu_0 \mathbf{1} \mathbf{1}^{\top}.
For kernel ridge regression (KRR), the filter is
For kernel gradient flow with time parameter \lambda^{-1}, the filter is
Let q \geq 1 be fixed. The q-fold iterated ridge estimator corresponds to the filter
Kernel gradient descent may be viewed as a time-discretized version of kernel gradient flow. Fix a step size \eta>0, and apply gradient descent to the empirical loss for t iterations. The resulting estimator is associated with the spectral filter
There exists an absolute constant c_{0} > 0 such that for any d \geq \mathfrak{C}, we have
Moreover, if s> 2\tau, we further assume that
Several lower bound results in the literature remain valid, in the sense that the same proofs apply, if their respective non-degeneracy conditions are replaced by Assumption 6. These include Lemmas 21–23 of [zhang2024optimal], Lemma D.14 of [lu2024saturation], and Lemmas 5.4.3–5.4.4 of [zhang2024phase].
Suppose Assumptions 1, 2, 3, 4, 5, and 6 hold with \tau \leq \infty, s \geq 0, and u \in (0, \infty]. Denote \tilde{s}=\min\{s, 2\tau\}. Then the excess risk of the spectral algorithm estimator in (?) satisfies
Theorem 3.1 reveals an interesting phenomenon. When s is sufficiently small, the error can remain dominated by the bias even for large u in the under-regularized and interpolation regimes. For example, consider the case \gamma = 0.9, s=0.5, and \tau=\infty. In this case, we have u^{\prime}=0.45. For any u \geq 0.45, the variance term is of order d^{\gamma - \tilde{\ell} - 1 - 2 \max\{\gamma - u, 0\}} + d^{\tilde{\ell} - \gamma} = o_d(d^{-(\tilde{\ell}+1)s}), so it is negligible relative to the bias term. This suggests that, when s is sufficiently small, spectral algorithms can still achieve strong performance, and even benign overfitting, well into the under-regularized and interpolation regime. Section ? makes this statement precise by characterizing benign overfitting.
Suppose all assumptions in Theorem 3.1 hold and define p := \lfloor \gamma/(s+1) \rfloor. Recall that u^{\prime} = u^{\prime}(\tau) is the optimal choice of u that balances the rates of the bias and variance terms (see Proposition F.4 for its expression). Then we have the following statements:
(i) Suppose 0 \leq s \leq \Gamma(\gamma). Then for any u \in [u^{\prime}, \infty], the excess risk of the spectral algorithm estimator in (?) satisfies
(ii) Suppose s > \Gamma(\gamma). Then there exists \tilde{u} \in (u^{\prime}, \gamma) such that for \tilde{\lambda}=d^{-\tilde{u}}, we have
Adopt all the notations and assumptions in Theorem 3.1.
Then the excess risk of (\hat{f}_j^{\lambda})_{j=1}^{\infty} satisfies
Consequently, Theorem 3.1 implies
We assume that there exist positive absolute constants c_1, c_2, and a non-integer \gamma>0, such that the sample size satisfies
c_{1} d^{\gamma} \leq n \leq c_{2} d^{\gamma},
and we denote \ell_{\gamma} := \lfloor \gamma\rfloor < \gamma.
We assume that the sum of the eigenvalues is bounded, that is, there exists an absolute constant \tilde{\mathfrak{C}} such that \sum_{k=0}^{\infty} \sum_{j=1}^{\tilde{N}(d, k)} \tilde{\lambda}_{k, j}\leq \tilde{\mathfrak{C}}. Moreover, for any k = 0, 1, \cdots, 2\ell_{\gamma} +2, we assume that
For any integer q \geq 1, there exists a constant C=C(q, \ell_{\gamma}) such that
[misiakiewicz2024non] also considers a hyper-contractivity assumption on the unknown true regression function f_{\star}, which posits that \left\|f_{\star}\right\|_{L^q} \leq (C q)^{(\ell_{\gamma}+1) / 2}\left\|f_{\star}\right\|_{L^2} for all integers q \geq 2. In Appendix I.5, we use a counterexample to demonstrate that the hyper-contractivity of eigen-functions does not imply the hyper-contractivity of the unknown true regression function. Therefore, the hyper-contractivity assumption on f_{\star} is not weaker than Assumption 9.
There exist absolute constants s \geq 0 and R>0, such that we have
Suppose there exists an absolute constant c_{0} > 0 such that for any d \geq \mathfrak{C}, we have
Moreover, if s> 2, we further assume that \sum\limits_{k=0}^{\tilde{\ell}} \sum\limits_{j =1}^{\tilde{N}(d, k)} \theta_{k, j}^{2} \ge c_{0}.
Suppose Assumptions 4, 7, 8, 9, 10, and 11 hold with s \geq 0 and u \in (0, \infty]. Denote \tilde{s}=\min\{s, 2\}. Then the excess risk of the KRR estimator in (?) satisfies
Consequently, results in Theorem 3.3 still hold under the above assumptions and conditions.
Two existing results [mei2022generalization, misiakiewicz2024non] also provide tight bounds on excess risk of large-dimensional KRR with general kernels defined on general domains. Specifically, [mei2022generalization] showed that
where \text{E}_{\text{approx}} is a certain deterministic quantity. However, as we have discussed in Appendix B.2, such results are uninformative when s>0 since the excess risk is o_{d, \mathbb{P}}(1). Later, [misiakiewicz2024non] developed a tighter approximation (see, e.g., Theorem 1 and Theorem 2 of [misiakiewicz2024non])
However, their assumptions are either difficult to verify (Assumption 1 of [misiakiewicz2024non]), or are too restrictive when s>0, as we have discussed in Appendix B.3. On the contrary, for kernel ridge regression methods, Theorem 4.2 not only extends Theorem 3.1 from kernels on the sphere to kernels on general domains. It also extends the results of [mei2022generalization] and [misiakiewicz2024non] by determining the convergence rate of the excess risk when s>0.
The condition (?) is very mild: the LHS is of order O_{d, \mathbb{P}}(d^{-1}). To see this, note that for any 1 \leq i \neq i^{\prime} \leq 3, the inner products \langle \xi_{i}, \xi_{i^{\prime}} \rangle have mean zero and variance (d+1)^{-1}, thus for any \varepsilon>0, from Chebyshev’s inequality we have \mathbb{P}(\sum_{i \neq i^{\prime}}^{3} \left| \langle \xi_{i}, \xi_{i^{\prime}} \rangle \right| \geq \varepsilon) \leq 6 \max_{1 \leq i \neq i^{\prime} \leq 3} \mathbb{P}( \left| \langle \xi_{i}, \xi_{i^{\prime}} \rangle \right| \geq \varepsilon) \leq 6/[\varepsilon^2(d+1)].
We select KGF and KRR for experiments due to their dual significance. On one hand, they are central and widely studied instances of spectral algorithms for kernel regression. On the other hand, the class of analytic spectral algorithms is characterized by a qualification parameter \tau \in [1, \infty], with KRR and KGF serving as canonical representatives of the two ends, \tau=1 and \tau =\infty. Hence, their agreement with the predicted slopes shows that the large-dimensional asymptotic picture in Theorem 3.1 is already visible at finite sample sizes for the two canonical endpoint filters, and provides empirical support for the broader theory of analytic spectral algorithms.
We list all the absolute constants used in this paper:
\sigma^2: Upper bound on variance of the noise in (1).
c_1, c_2, \gamma > 0: Positive constants in the asymptotic framework (2).
a_{-1}; a_0, a_1, \ldots, a_{\lfloor \gamma \rfloor + 2} > 0 : The sum and the first (\lfloor \gamma \rfloor+ 3) coefficients of the Taylor expansion of \Phi(\cdot) as specified in Assumption 2.
s \geq 0, R>0: Coefficients in the source condition (6).
c_{0} >0: A constant given in Assumption 6.
C_i > 0, i=1, \cdots, 4: The constants related to the filter function defined in Definition 2.2.
\tau^{\prime}: A finite constant defined in (?).
\Delta u :=(u - \ell_{\lambda})\mathbf{1}\{u<\infty\} \in [0, 1): u is the decay rate of the regularization parameter \lambda in Assumption 4, and \ell_{\lambda} = \lfloor u \rfloor.
c_3, c_4 > 0: Positive constants in Assumption 4.
\tilde{\ell}=\min\{\ell_{\gamma}, \ell_{\lambda}\} defined in Assumption 4, where \ell_{\gamma} = \lfloor \gamma \rfloor.
Let u^{\prime} = u^{\prime}(\tau) be the rate of the optimal regularization parameter \lambda with qualification \tau \leq \infty and s>0 be the coefficient of source condition in [lu2024saturation] (ignoring the logarithm term). Suppose Assumptions 1,2,3,4,5, and 6 hold with \tau \leq \infty, s>0, and u \in (0, u^{\prime}). Denote \tilde{s}=\min\{s, 2\tau\}. Further, suppose at least one of the following conditions hold:
Then the excess risk of the spectral algorithm estimator in (13) satisfies
Let \mathcal{P} consist of all the distributions \rho_{f_{\star}} on \mathcal{X} \times \mathcal{Y} given by (1) such that Assumption 1,2, and 3 hold for some \gamma, s>0. Furthermore, assume that there exists an absolute constant \alpha>0, such that we have n = \alpha d^{\gamma} (1+o_{d}(1)). Define p := \lfloor \gamma/(s+1) \rfloor. Then, we have
where \hat{f} is any estimator of f_{\star}, measurable with respect to the observed data set (X, Y), and the definition of the Pinsker constant \mathcal{C}^{\star} can be found in Theorem 3.1 in [lu2024pinsker].
Let \hat{f}_{\mathrm{inter}}(\cdot):= \hat{f}_{0}(\cdot) = \mathscr{K}(\cdot, X) \left(\mathscr{K}(X, X)\right)^{-1} Y be the kernel interpolation estimator. Suppose Assumptions 1,2,3, and 6 hold with s \geq 0. Denote \tilde{s}=\min\{s, 2\tau\}. Then the excess risk of the kernel interpolation estimator satisfies
Notice that the above sequence model is defined implicitly, since the variance of the noise depends on the excess risk, which itself depends on the noise.
Suppose Assumption 1 and 2 hold for some \gamma>0. Further suppose there exists an absolute constant C, such that we have \left\|\mathrm{P}_{>\ell_{\gamma}} f_{\star}\right\|_{L^2} \geq \left\|f_{\star}\right\|_{L^2} / C, and for all integers q \geq 2, we have \left\|f_{\star}\right\|_{L^q} \leq (C q)^{(\ell_{\gamma}+1) / 2}\left\|f_{\star}\right\|_{L^2}. Then under certain conditions, we have
where \mathrm{P}_{>\ell_{\gamma}} means the projection onto spherical harmonics with degree >\ell_{\gamma}, and E_{\tilde{\xi}} means taking expectation on the noises \tilde{\xi}_j, j=1, 2, \cdots.
The assumptions in Proposition B.5 are pivotal to its proof technique and are intrinsically difficult to relax. Notice that the two assumptions,
are sufficient conditions for the convex concentration on eigenfunctions in [cheng2022dimension], which allows them to apply the Hanson-Wright inequality in Lemma 2.1 of [cheng2022dimension]. In contrast, our proof employs a different analytical framework that does not rely on those restrictive premises.
Suppose Assumptions 1, 2, and 4 hold. Then the variance term (14) of the spectral algorithm estimator \hat{f}_{\lambda} satisfies
Because \mathscr{K}(x,X) = \psi(x)^{\top} \Sigma \Psi^{\top} and that \epsilon_1, \cdots, \epsilon_n are independent with variance \sigma^2, the variance term (14) can be expressed as
where \kappa_{2} = \Theta_{d}\left( d^{-(\ell_{\gamma}+1)} \right) and \|\Delta_{2}\|_{\mathrm{op}} = o_{d, \mathbb{P}}(1). We then bound \mathrm{tr}(V_1) and \mathrm{tr}(V_2) separately.
Bounding \mathrm{tr}(V_1). On the one hand, we have
On the other hand, we have
where the third inequality (\ast) can be shown using the Jensen’s inequality as in Remark ?. Therefore, \mathrm{tr}(V_1) = \Theta_{d, \mathbb{P}}\left(\frac{n}{d^{\ell_{\gamma}+1}(n\lambda+1)^2}\right).
Suppose A = \text{diag}(a_1, \cdots, a_n) \in \mathbb{R}^{n \times n} is a positive diagonal matrix. Then, the Jensen’s inequality f\left( \frac{a_1 + \cdots + a_n}{n} \right) \leq \frac{f(x_1) + \cdots + f(x_n)}{n} for convex function f(x) = x^{-2}, x>0 implies
Since M is positive definite, we can show (\ast) with eigendecomposition.
Bounding \mathrm{tr}(V_2). We bound \mathrm{tr}(V_2) in two cases: (i) \ell_{\gamma} \leq \ell_{\lambda}, and (ii) \ell_{\gamma} > \ell_{\lambda}.
(i) On the one hand, when \ell_{\gamma} \leq \ell_{\lambda}, we have
where the last equality follows from Lemma G.2 and G.3, together with the fact that N_{\ell_{\gamma}}:= \sum_{k=0}^{\ell_{\gamma}} N(d, k) = \Theta(d^{\ell_{\gamma}}) (see Proposition 2.1).
(ii) On the other hand, when {\ell_{\gamma}} > \ell_{\lambda}, similar to (?), we have
hence
For the upper bound of \mathrm{tr}(V_{2, 1}), we have
Next, we give the lower bound of \mathrm{tr}(V_{2, 1}). From Lemma G.2 and G.3, we can denote the SVD decomposition of \Psi_{\ell_{\lambda}+1} as \Psi_{\ell_{\lambda}+1} = n^{1/2} ~ O H V^{\top}, where O \in \mathbb{R}^{n \times n} and V \in \mathbb{R}^{N(d, \ell_{\lambda}+1) \times N(d, \ell_{\lambda}+1)} are orthogonal matrices; and HH^{\top}=\text{Diag}\{h_1^2 \geq \cdots \geq h_n^2 \geq 0\} is a diagonal matrix with h_1^2=O_{d, \mathbb{P}}(1) and
Denote the SVD decomposition of \varphi_{\lambda, K}^2 as \varphi_{\lambda, K}^2=PDP^{\top}, where D=\text{Diag}\{d_1 \geq \cdots \geq d_n > 0\} is a diagonal matrix.
Since \ell_{\gamma} = \lfloor \gamma \rfloor > \ell_{\lambda}, N_{\ell_{\lambda}} = \Theta(d^{\ell_{\lambda}}), and N(d, \ell_{\lambda}+1)= \Theta(d^{\ell_{\lambda}+1}), without loss of generality we assume that
Recall that we denote HH^{\top}=\text{Diag}\{h_1^2 \geq \cdots \geq h_n^2 \geq 0\} and D=\text{Diag}\{d_1 \geq \cdots \geq d_n > 0\}. Therefore, we have
where the third line comes from von Neumann’s trace inequality (see also Proposition K.2).
Combining all the above, we have
and we get the desired results.
Suppose Assumptions 1–6 hold. Then, the bias term (15) of the spectral algorithm estimator \hat{f}_{\lambda} satisfies:
where
(i) We have
(ii) If 0 < u < \gamma <1, then for any \tau \leq \infty, we have
where 0 \leq \Delta_1 = O_{d, \mathbb{P}}\left( d^{-2\gamma }\right);
(iii) We have
(iv) We have
(v) We have
We remark that the upper bound in Theorem E.1 is not tight in certain cases, for which we will provide a tight bound in Proposition E.3.
We consider the following two cases: (I) we have \gamma > \ell_{\gamma}, or we have \ell_{\lambda} < \gamma = \ell_{\gamma}, or we have \ell_{\lambda} \geq \gamma = \ell_{\gamma} and n \geq 2N(d, \ell_{\gamma});
(II) we have \ell_{\lambda} \geq \gamma = \ell_{\gamma} and n < 2N(d, \ell_{\gamma}).
paragraph4 1.5ex 0.5ex 0.2ex -1em 11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt Case (I). Suppose that we have \gamma > \ell_{\gamma}, or we have \ell_{\lambda} < \gamma = \ell_{\gamma}, or we have \ell_{\lambda} \geq \gamma = \ell_{\gamma} and n \geq 2N(d, \ell_{\gamma}). We first prove that
Eq. (?) can be seen by considering the following:
Suppose \ell_{\lambda} \geq \ell_{\gamma}. Since \tilde{\ell}=\ell_{\gamma}, from Lemma G.2 and Lemma G.3, we have Eq. (?) holds.
Suppose \ell_{\lambda}<\ell_{\gamma}<\gamma. Recall that \Psi_{\leq \ell_{\gamma}} = \left[\Psi_{\leq \ell_{\lambda}}, \Psi_{\ell_{\lambda}+1, \ell_{\gamma}}\right]. Since \tilde{\ell}=\ell_{\lambda}, from Lemma G.2, we have
which implies Eq. (?).
Suppose \ell_{\lambda}<\ell_{\gamma} = \gamma. Similar to the last case, we can use Lemma G.3 to show Eq. (?).
The bias term Eq. (15) can be decomposed as
Before we bound these terms separately, we need the following calculations.
Denote Z = n^{-1/2}\Psi_{\leq \tilde{\ell}}.
Suppose the SVD decomposition of Z is Z=U (H, 0)^{\top} V^{\top}, where U \in \mathbb{R}^{n \times n}, V \in \mathbb{R}^{N_{\tilde{\ell}} \times N_{\tilde{\ell}}} are orthogonal matrices, and H = \text{Diag}\{h_1, \cdots, h_{N_{\tilde{\ell}}}\} is a diagonal matrix. Since Eq. (?) holds, we have h_1, \cdots, h_{N_{\tilde{\ell}}} = \Theta_{d, \mathbb{P}}(1).
Hence,
where we have defined the following
Bounding \|B_1\|_{2}^{2}. We consider the following two cases: (B1.i) \ell_{\gamma} \leq \ell_\lambda; (B1.ii) \ell_{\gamma} > \ell_\lambda.
Proof of case (B1.i).
When \ell_{\gamma} \leq \ell_\lambda,
from (?) we have
For B_{1,1}:
From Lemma G.1, there exist constants \mathfrak{C}_5, \mathfrak{C}_6 >0 such that we have \mathfrak{C}_5 \mathrm{I}_n \leq K_{>\tilde{\ell}} \leq \mathfrak{C}_6 \mathrm{I}_n. Furthermore, \tilde{\ell}=\ell_{\gamma} since \ell_{\gamma} \leq \ell_{\lambda}, and \lambda_{\min}(\Sigma_{\leq \ell_{\gamma}})=\mu_{\ell_{\gamma}}=\Omega(d^{-\ell_{\gamma}}) by Proposition 2.1. Hence, we have \lambda_{\max}(K_{>\tilde{\ell}}/n)=O_{d, \mathbb{P}}\left(\lambda_{\min}(\Sigma_{\leq \tilde{\ell}})\right).
Therefore, from (?) we have
For B_{1,2}:
Eq. (?) implies that \lambda_{\min}(Z^{\top}Z)=\Theta_{d,\mathbb{P}}(1)) and \lambda_{\max}(Z)=\Theta_{d,\mathbb{P}}(1). Therefore,
Therefore,
where
the second equality follows from Eq. (?);
the third equality is because \lambda_{\max}(Z)=\Theta_{d,\mathbb{P}}(1), ~\lambda_{\min}(Z^{\top}Z)=\Theta_{d,\mathbb{P}}(1);
the last equality follows from Lemma G.7(iii).
Proof of case (B1.ii). Suppose \ell_\lambda < \ell_{\gamma}.
Let A=Z \Sigma_{\leq \tilde{\ell}} Z^{\top} and B=K_{>\tilde{\ell}} / n. Then A+B=Z \Sigma_{\leq \tilde{\ell}} Z^{\top}+K_{>\tilde{\ell}} / n=K/n. By Proposition J.1, we have
It follows that
where the first equation follows from the 4th line in Equation (?), and the first inequality is because of Equation (?) and \lambda_{\min}(Z^{\top}Z)=\Theta_{d,\mathbb{P}}(1)). By the same reasoning as in Eq. (?), we can show that B_{1,3} = O_{d, \mathbb{P}}\left( \lambda^{2} d^{(2-\min\{s, 2\})\tilde{\ell} }\right). Moreover, we have
where the last equation follows from Lemma G.7 (iii).
Finally, denote \tilde{Z}=n^{-1/2}\Psi_{\ell_{\lambda}+1, \ell_{\gamma}}. Recall that K_{\leq \tilde{\ell}}=Z\Sigma_{\leq \tilde{\ell}} Z^{\top}. Using K^{-1}K_{\leq \tilde{\ell}}=\mathrm{I}_n-K^{-1}K_{>\tilde{\ell}}, we have
It suffices to show that B_{1, 6}=o_{d,\mathbb{P}}(n^{-1}).
We will use the matrix Bernstein inequality to bound B_{1, 6} as follows.
For i=1,\cdots, n, denote
By the addition formula (see, e.g., Proposition 1.18 in [gallier2009notes], page 56 in [ghorbani2021linearized], or (39) in [lu2024pinsker]), we have
and
Therefore, we have B_{1, 6} = \|\sum\nolimits_{i=1}^{n}A_i\|_{2}^{2}, and we can compute \mathbb{E}A_i =~ 0,
and similarly
black Therefore, from Proposition K.3, we have
and hence B_{1, 6} = \|\sum\nolimits_{i=1}^{n}A_i\|_{2}^{2} = o_{d, \mathbb{P}}(n^{-1}).
Proof of the case 0 \leq u < \gamma <1.
If 0 \leq u < \gamma <1, we have \ell_{\gamma}=\ell_{\lambda}=0, from (?), we have
Following the same derivation in Eq. (?) but with the triangle inequality applied in an opposite direction, when \tau < \infty we have
where the third last and the second last lines use the fact that \Sigma_{\leq 0} = \mu_0 = \Theta_d(1) obtained by Proposition 2.1, and (?) implies 0 \leq \Delta_1 = \left\| B_{1, 2} \right\|_{2}^{2} = O_{d, \mathbb{P}}\left( d^{-2\gamma }\right).
Similarly to (?) and above, for any \tau \leq \infty, we have
In summary, we have completed the proof for the following statements about B_{1}:
We have
If 0 < u < \gamma <1, then for any \tau \leq \infty, we have
where 0 \leq \Delta_1 = O_{d, \mathbb{P}}\left( d^{-2\gamma }\right).
Bounding \|B_2\|_{2}^{2}. We have
Bounding \|B_3\|_{2}^{2}. We have
Bounding \|B_4\|_{2}^{2}. We have
We derive the bound on \left\| K_{\leq \tilde{\ell}} \varphi_{\lambda, K} K_{> \tilde{\ell}} \varphi_{\lambda, K} K_{\leq \tilde{\ell}}\right\|_{2} in two cases.
Case 1: When \tilde{\ell} = \ell_{\gamma} \leq \ell_{\lambda}, we have u\geq \gamma and
Case 2: When \ell_{\gamma} > \ell_{\lambda}, we have
Combining both cases, we have
Bounding \|B_5\|_{2}^{2}. We have
paragraph4 1.5ex 0.5ex 0.2ex -1em 11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt11.5pt13.2pt Case (II). When \ell_{\lambda} \geq \gamma = \ell_{\gamma} and n < 2N(d, \ell_{\gamma}),
denote {N}_{\underline{\ell_{\gamma}}}=\sum_{k=0}^{\ell_{\gamma}-1}N(d,k)+n/4 < N_{\ell_{\gamma}} and
then, Lemma G.3 implies that
Similar to Appendix ?, we can denote \Psi_{> {\underline{\ell_{\gamma}}}}, \Sigma_{\leq {\underline{\ell_{\gamma}}}}, \Sigma_{> {\underline{\ell_{\gamma}}}}, \theta_{\leq {\underline{\ell_{\gamma}}}}, \theta_{> {\underline{\ell_{\gamma}}}}, K_{\leq {\underline{\ell_{\gamma}}}}, and K_{> {\underline{\ell_{\gamma}}}}.
Notice that we have {N}_{\underline{\ell_{\gamma}}} = \Theta_d(n)=\Theta_d(d^{\ell_{\gamma}}) = \Theta_d(N(d, \ell_{\gamma})) = \Theta_d(N_{\ell_{\gamma}}), hence from Lemma G.3, similar to the proof of (iv) in Lemma G.6, we have
Similarly, we can show that Lemma G.6(v), Lemma G.7, and Lemma G.8 hold with \tilde{\ell} replaced by \underline{\ell_{\gamma}}. Therefore, following the above proof for Case (I), we can obtain the same bounds for \|B_i\| with i=1, \cdots, 5.
Suppose Assumptions 1–6 hold. Suppose that \tau \leq \infty, s>0,
u^{\prime}(\tau) \leq u < u^{\prime}(1), and \gamma \geq 1. Then, the bias term (15) of the spectral algorithm estimator \hat{f}_{\lambda} satisfies:
where 0 \leq \Delta_2 = o_{d, \mathbb{P}} \left( d^{\gamma - \tilde{\ell} - 1 - 2 \max\{\gamma - u, 0\}} + d^{\tilde{\ell} - \gamma} + d^{-(\tilde{\ell} + 1) s} + \mathbf{1}\{\tau<\infty\} d^{-2 \tau u + (2 \tau - \tilde{s}) \tilde{\ell}} \right).
We adopt notations in [lu2024saturation], and especially:
We also note that the following notations are different in our paper and in [lu2024saturation, zhang2024optimal]:
In [lu2024saturation], \ell:=\lfloor u \rfloor; while in our paper, we denote \ell_{\lambda}:=\lfloor u \rfloor.
Consider the case where 0<s<1. In this case, \tilde{s}:=\min(s, 2\tau)=s, so the notation \tilde{p} = \lfloor \gamma / (\tilde{s}+1) \rfloor used in [zhang2024optimal] and in Proposition F.4 is the same as the notation p = \lfloor \gamma / (s+1) \rfloor in [lu2024saturation]. Notice that one can show that p= \tilde{p} < \gamma (see Proposition F.4).
Without loss of generality, we assume that \lambda=d^{-u}. We divide the proof into the following three cases: (i) 1 \leq s \leq \tau, (ii) \tilde{s}:=\min\{s, 2\tau\} > \tau, and (iii) 0<s<1.
Case (i): 1 \leq s \leq \tau. Notice that:
From (36) and (41) of [lu2024saturation], we have that \mathrm{bias}(\hat{f}_{\lambda}) equals to the bias term defined in (41) of [lu2024saturation].
From (60) of [lu2024saturation], we have that \mathrm{bias}(\hat{f}_{\lambda}) \leq \left\| f_{\lambda} - f_{\star}\right\|_{L^{2}} + \left\| \tilde{f}_{\lambda} - f_{\lambda}\right\|_{L^{2}} and \mathrm{bias}(\hat{f}_{\lambda}) \geq \left\| f_{\lambda} - f_{\star}\right\|_{L^{2}} - \left\| \tilde{f}_{\lambda} - f_{\lambda}\right\|_{L^{2}}. Moreover, the latter two terms can be upper bounded separately:
From Lemma D.6 and Lemma D.14 in [lu2024saturation], we have
Suppose the following inequalities hold (which we will prove later)
Then, on one hand, since u < u^{\prime}(1) < \gamma, from Lemma 23 in [zhang2024optimal], there exists \epsilon>0, such that
On the other hand, following the proof of Lemma D.7 in [lu2024saturation], with (63) and (64) of [lu2024saturation] being
restated as (?), we have
Proposition F.4 implies that u < u^{\prime}(1) < \gamma, hence \tilde{\ell}=\ell_{\lambda}.
Therefore, from the above arguments, we only need to verify that (?) hold.
We remark that the above argument is the same as that in [lu2024saturation], where they verify an analogous version of (?) via their Lemmas D.21-D.23.
Since \tau \geq s, we have (\lambda d^{\ell_{\lambda}})^{\tau} \leq (\lambda d^{\ell_{\lambda}})^{s}, hence from Lemma D.14 in [lu2024saturation] we have
Denote \mathbf{I}=\lambda^{2(s-1)} d^{-\gamma+\ell_{\lambda}s}, \mathbf{II}=\lambda^{-1} d^{-\gamma -(\ell_{\lambda}+1)(s-1)}, \mathbf{III}=\lambda d^{\max\{\ell_{\lambda}(2-s), 0\}}(\ln(d))^3,
and \mathbf{IV}=\lambda^{-1}d^{-s(\ell_{\lambda}+1)}(\ln(d))^3. Denote A_1 = d^{2u-\ell_{\lambda}-1}, A_2 = d^{\ell_{\lambda}}, A_3 = d^{\gamma-s(\ell_{\lambda}+1)}. Then, (?) is implied by the following relationships:
Recall that Proposition F.4 implies that u < u^{\prime}(1) < \gamma. Hence, from Lemma D.14 in [lu2024saturation], we have that
We have
Proposition F.4 implies that u < u^{\prime}(1) < \gamma, hence
When s\geq 2, we have
When s<2, we have
Proposition F.4 implies that u < u^{\prime}(1) < \gamma, hence
Combining all these, we obtained (?).
Case (ii): \tilde{s}:=\min\{s, 2\tau\} > \tau. Similar to Case (i), we only need to show that (?) holds. From Lemma D.14 in [lu2024saturation] we have
Denote \mathbf{I}=\lambda^{2(\tau-1)} d^{-\gamma+\ell_{\lambda}(2\tau-\tilde{s})}, \mathbf{II}=\lambda^{-1} d^{-\gamma -(\ell_{\lambda}+1)(s-1)}, \mathbf{III}=\lambda d^{\max\{\ell_{\lambda}(2-s), 0\}}(\ln(d))^3,
and \mathbf{IV}=\lambda^{-1}d^{-s(\ell_{\lambda}+1)}(\ln(d))^3. Denote A_1 = d^{2u-\ell_{\lambda}-1}, A_2 = d^{\ell_{\lambda}}, A_3 = d^{\gamma-s(\ell_{\lambda}+1)}. Then, (?) is implied by the following relationships:
Recall that Proposition F.4 implies that u < u^{\prime}(1) < \gamma. Hence, from Lemma D.14 in [lu2024saturation], we have that
When s >2\tau, we have
When \tau < s \leq 2\tau, we have (\lambda d^{\ell_{\lambda}})^{2\tau} \leq (\lambda d^{\ell_{\lambda}})^{s}, hence
Same as the proof in Case (i), we have
Combining all these, we obtained (?).
Case (iii): 0< s <1. The proof is based on the following argument, which is similar to the one in Case (i):
Suppose there exists a constant \epsilon>0 only depending on s and \gamma, such that the following inequalities hold (which we will prove later)
Then, following the proof of Lemma D.8 in [lu2024saturation], where (75) and (76) therein are restated as (?), we obtain the desired bounds for the bias term \mathrm{bias}^{2}(\hat{f}_{\lambda}).
Therefore, we only need to establish (?).
Proposition F.4 implies that u < u^{\prime}(1) < \gamma, hence from Lemma D.14 in [lu2024saturation] and Lemma 23 in [zhang2024optimal], we have
Moreover, notice that (8) implies \varphi_\lambda(z) \leq C_2 (z+\lambda)^{-1}. Hence, similar to the proof of Lemma 27 in [zhang2024optimal] (for kernel ridge regression estimator), we can show that
Denote \mathbf{I}=\lambda d^{\ell_{\lambda}(2-s)} (\ln(d))^3, \mathbf{II}=\lambda^{-1}d^{-s(\ell_{\lambda}+1)}(\ln(d))^3, \mathbf{III}=\lambda^{-1} d^{-\gamma s}, \mathbf{IV}= d^{u-\gamma + \frac{(1-s)\ell_{\lambda}}{2}}, and \mathbf{V}= d^{2u-\gamma -\frac{(1+s)(\ell_{\lambda}+1)}{2}}. Denote A_1 = d^{2u-\ell_{\lambda}-1}, A_2 = d^{\ell_{\lambda}}, A_3 = d^{\gamma-s(\ell_{\lambda}+1)}.
Note that (?) is implied by the following inequalities:
Recall that Proposition F.4 implies that u < u^{\prime}(1) < \gamma. Hence, from Lemma D.14 in [lu2024saturation], we have that
We have
Proposition F.4 implies that u < u^{\prime}(1) < \gamma, hence
For \mathbf{III}, we analyze three possible cases:
Suppose u-\ell_{\lambda} < \gamma s. Then
Suppose u-\ell_{\lambda} \geq \gamma s and s>1/2. Then
Suppose u-\ell_{\lambda} \geq \gamma s and s \leq 1/2. We first prove by contradiction that \ell_{\lambda}<\ell_{\gamma}: Suppose that \ell_{\lambda} = \ell_{\gamma}, then from Case III in the proof of Proposition F.4 with \tau=1 and \tilde{p}=p = \lfloor \gamma / (s+1) \rfloor, we have
implying that \ell_{\lambda}=p. Hence, from Case III in the proof of Proposition F.4 with \tilde{p}=p, we further have
leading to a contradiction. (Notice that p(s+1) \leq \gamma < ps+p+s with p=0 is not allowed since \gamma \geq 1 >s). Therefore, we have \ell_{\lambda}<\ell_{\gamma}, and hence
We have
Finally, we have
Combining all the above steps, we obtained (?).
Suppose Assumptions 1,2,3,4,5, and 6 hold with \tau \leq \infty and s=0. Then, the excess risk of the spectral algorithm estimator in (13) satisfies
Notice that b_1=0 and these four quantities are non-positive: v_1, v_2, b_2(u, 1), b_3. Hence (?) implies that
and that
and we obtain the desired results.
Suppose Assumptions 1,2,3,4,5, and 6 hold with s>0. Denote \tilde{s}=\min\{s, 2\tau\}. Assume in addition that either \tau=\infty, or s>1/(2\tau), or \gamma > ((2\tau+1)s) / (2\tau(1+s)). Then the excess risk of the spectral algorithm estimator in (13) satisfies
Let’s separate the proof into four parts:
(i) u < u^{\prime}(\tau);
(ii) u^{\prime}(\tau) \geq u^{\prime}(1) and u \geq u^{\prime}(\tau);
(iii) u^{\prime}(\tau) < u^{\prime}(1) and u \geq u^{\prime}(1);
(iv) u^{\prime}(\tau) < u^{\prime}(1) and u \in [u^{\prime}(\tau), u^{\prime}(1)).
Case (i). Suppose u < u^{\prime}(\tau). (?) implies
Case (ii). Suppose u^{\prime}(\tau) \geq u^{\prime}(1). For any u \geq u^{\prime}(\tau) \geq u^{\prime}(1), from Proposition F.5 and Proposition F.6, we have
Therefore from (?) we have
Upper bound:
Lower bound:
Case (iii). Suppose u^{\prime}(\tau) < u^{\prime}(1). For any u \geq u^{\prime}(1) \geq u^{\prime}(\tau), from Proposition F.5 and Proposition F.6, we have
Therefore, as in Case (ii), we can show that
Case (iv). Suppose u^{\prime}(\tau) < u^{\prime}(1) and u \in [u^{\prime}(\tau), u^{\prime}(1)). From Proposition F.5 and Proposition F.6, we have
Therefore,
(a) When \gamma \geq 1, from the variance bound in (?) and the bias bound in (?), we have
(b) When \gamma<1 and \tau<\infty, Proposition F.4 with t=1 further implies 0<u<u^{\prime}(1) <\gamma<1. Hence from (?) and (?) we have
Upper bound:
Lower bound:
If b_1>b_4, then
If b_1 \leq b_4, since Proposition F.5 implies that b_4<v_2, hence (?) implies
Therefore,
(c) Finally, suppose \gamma<1 and \tau = \infty. Recall that from (?), we have
Therefore, from Proposition F.4 we have
implying that u^{\prime}(\infty) \geq u^{\prime}(\tau^{\prime}).
Furthermore, the assumption of Case (iv) implies u \geq u^{\prime}(\infty)\geq u^{\prime}(\tau^{\prime}). Therefore, from Proposition F.5 and Proposition F.6, we have
From the assumption of Case (iv), we have u<u^{\prime}(1). From Proposition F.4 we have 0<u<u^{\prime}(1) <\gamma<1.
Hence from (?) and (?) we have
Upper bound:
Lower bound:
If b_1>b_4, then
If b_1 \leq b_4, since Proposition F.5 implies that b_1 \leq b_4<v_2, hence we have
Combining all above cases, we obtain the desired results.
Suppose Assumptions 1,2,3,4,5, and 6 hold with s>0. Denote \tilde{s}=\min\{s, 2\tau\}. Assume in addition that both \tau<\infty, and s \leq 1/(2\tau), and \gamma \leq ((2\tau+1)s) / (2\tau(1+s)). Then, the excess risk of the spectral algorithm estimator in (13) satisfies
Let’s separate the proof into two parts:
(i) u < \gamma;
(ii) u \geq \gamma.
Case (i). Suppose u < \gamma. Since s \leq 1/(2\tau) and
hence we have 0 < u < \gamma <1. Therefore,
from (?) and (?) we have
Upper bound:
Lower bound:
If b_1>b_4 and b_1 \geq b_2(u, \tau), then
If b_1 \leq b_4 and b_1 \geq b_2(u, \tau), since Proposition F.5 implies that b_4<v_2, we have \max\{b_1, b_2(u, \tau)\}<v_2, hence
If b_2(u, \tau)>b_3 and b_1 < b_2(u, \tau), then
If b_2(u, \tau) \leq b_3 and b_1 < b_2(u, \tau), since Proposition F.5 implies that b_3 \leq v_2, we have \max\{b_1, b_2(u, \tau)\} \leq v_2, hence
Case (ii). Suppose u \geq \gamma. From Proposition F.4 with t=1 and t=\tau, we have \gamma > u^{\prime}(1) and \gamma > u^{\prime}(\tau), hence from Proposition F.5 and Proposition F.6, we have
Therefore, from (?) we have
Upper bound:
Lower bound:
Putting the two bounds together yields the desired result.
Let u^{\prime} = u^{\prime}(t) be the optimal rate of the regularization parameter \lambda with qualification t \in [1, \infty] and coefficient of source condition s>0 defined in [lu2024saturation] (ignoring the logarithm term). Then, we have u^{\prime} < \gamma and \tilde{p} \leq u^{\prime} < \tilde{p}+1, where \tilde{s} := \min\{s, 2t\} and \tilde{p}:=\lfloor \gamma/(\tilde{s}+1)\rfloor.
As a direct corollary, for any u \leq u^{\prime}, we have \ell_{\lambda} = \lfloor u \rfloor \leq \ell_{\gamma}, hence
According to Corollaries D.15–17 in [lu2024saturation], we divide the proof into four cases.
Case I: 1 \leq s \leq t
Denote p:=\lfloor \gamma/(s+1)\rfloor.
(1.1) Suppose p \geq 1 and p(s+1) \leq \gamma < ps+p+s. We have
(1.2) Suppose p \geq 1 and ps+p+s \leq \gamma < ps+p+s+1. We have
(1.3) Suppose \gamma < s. We have
(1.4) Suppose s \leq \gamma < s+1. We have
Case II: \tilde{s} > t
Denote \tilde{p}:=\lfloor \gamma/(\tilde{s}+1)\rfloor.
(2.1) Suppose \gamma \geq 1 and 0 \leq \gamma - \tilde{p}(\tilde{s} +1) < t. We have
(2.2) Suppose \gamma \geq 1 and t \leq \gamma - \tilde{p}(\tilde{s} +1) < \tilde{s} + \tilde{s} /t - 1. We have
and \tilde{p} \leq u^{\prime} < \tilde{p}+1.
(2.3) Suppose \gamma \geq 1 and \tilde{s} + \tilde{s} /t - 1 \leq \gamma - \tilde{p}(\tilde{s} +1) < \tilde{s} +1. We have
(2.4) Suppose \gamma < 1. We have
Case III: t < \infty and s<1
Denote p:=\lfloor \gamma/(s+1)\rfloor.
(3.1) Suppose p(s+1) \leq \gamma < ps+p+s. We have
and p \leq u^{\prime} < p+1.
(3.2) Suppose ps+p+s \leq \gamma < ps+p+s+1. We have
Case IV: t = \infty and s<1
Denote p:=\lfloor \gamma/(s+1)\rfloor.
(4.1) Suppose p \geq 1 and p(s+1) \leq \gamma < ps+p+s. We have
(4.2) Suppose p \geq 1 and ps+p+s \leq \gamma < ps+p+s+1. We have
(4.3) Suppose \gamma < s. We have
(4.4) Suppose s \leq \gamma < s+1. We have
For any u \in (0, \infty], we have b_3(u) \leq v_2(u) and b_4(u) < v_2(u).
We have
Let u^{\prime} = u^{\prime}(t) be the optimal rate of the regularization parameter \lambda with qualification t < \infty and coefficient of source condition s>0 defined in [lu2024saturation] (ignoring the logarithm term). Then, for any u \geq u^{\prime}, we have
For notational simplicity, we denote \tilde{s}:=\min\{s, 2t\} and b_2(u)=b_2(u, t) for any u \in [0, \infty]. We divide the proof into three steps:
(i) We first show that for any u^{\prime} \leq u < \lceil u^{\prime} \rceil, we have
(ii) We then show that we have
(iii) Finally, we show that, for any integer q \geq 0 and any u\geq q, if \max\{v_1(q), v_2(q), b_1(q)\} \geq b_2(q), then we have
Combining (i), (ii), and (iii) with q=\lceil u^{\prime} \rceil, we obtain the desired result.
Proof of (i). For any integer q \geq 0 and any q \leq q_1 \leq q_2 < q+1, we have
Therefore, for any u^{\prime} \leq u < \lceil u^{\prime} \rceil, we have
Proof of (ii). If u^{\prime} is an integer, then from (i) we have \max\{v_1(\lceil u^{\prime} \rceil), v_2(\lceil u^{\prime} \rceil), b_1(\lceil u^{\prime} \rceil)\} \geq b_2(\lceil u^{\prime} \rceil).
Suppose u^{\prime} is not an integer. From Proposition F.4, we have \tilde{p} \leq u^{\prime} < \tilde{p}+1, where \tilde{p}:=\lfloor \gamma/(\tilde{s}+1)\rfloor. Denote \tilde{\ell}_{u^{\prime}} := \min\{\ell_{\gamma}, \tilde{p}+1\}, then we have \lceil u^{\prime} \rceil = \tilde{p}+1 and
If \gamma < \tilde{p}+1, then by the definition of \tilde{p} \leq \ell_{\gamma} = \lfloor \gamma \rfloor, we have \tilde{p}=\ell_{\gamma} and \tilde{\ell}_{u^{\prime}} = \ell_{\gamma}. Hence
If \gamma \geq \tilde{p}+1, then \tilde{\ell}_{u^{\prime}} = \tilde{p}+1. Since \gamma<(\tilde{p}+1)(\tilde{s}+1), we have
Combining the above two cases, we have
Proof of (iii). Suppose that for some integer q \geq 0, we have \max\{v_1(q), v_2(q), b_1(q)\} \geq b_2(q). By (i), for any q^{\prime \prime}\in[q, q+1) we have
Thus, the inequality holds throughout the interval [q, q+1).
Next, consider the integer point q+1. We analyze three possible cases:
If s \leq 2 t, then \tilde{s}=s and (?) implies
If s>2t and q < \ell_{\gamma}, then from (?) we have b_1(q)=-s(q+1) < -2t q = b_2(q), so the inequality \max\{v_1(q), v_2(q), b_1(q)\} \geq b_2(q) must come from the first two components:
Therefore, (?) implies
If s>2t and q \geq \ell_{\gamma}, then (?) implies
Combining (?) and all three cases above, we conclude that if
holds at integer u=q, it also holds for all u \in [q, q+1].
Finally, by induction over integers q+1, q+2, \cdots, we have
which completes the proof of (iii).
We have
with
When \gamma > \ell_{\gamma}, we have
Suppose \gamma = \ell_{\gamma}. Denote N_{\ell_{\gamma}-1} = \sum_{k^{\prime}=0}^{\ell_{\gamma}-1} N(d,k^{\prime}), then
And we have
hence
For completeness, we briefly explain how to obtain Lemma G.2 and Lemma G.3 from Lemma 11 in [ghorbani2021linearized] and Theorem 1 in [xiao2022precise]. A comparison of notations is given in Table ?. Recall
When \gamma > \ell_{\gamma}, Proposition 2.1 implies that n/(N_{\ell_{\gamma}} \log(N_{\ell_{\gamma}})) = \Theta_d(d^{\gamma - \ell_{\gamma}}\log^{-1}(d)) \to \infty as d \to \infty. Denote \Delta = \frac{\Psi_{\leq \ell_{\gamma}}^{\top} \Psi_{\leq \ell_{\gamma}} }{n} - \mathrm{I}_{N_{\ell_{\gamma}}}, then Lemma 11 in [ghorbani2021linearized] implies \mathbb{E}[\|\Delta\|_{\mathrm{op}}] = o_{d}(1), and Markov’s inequality implies \|\Delta\|_{\mathrm{op}} = o_{d, \mathbb{P}}(1).
When \gamma = \ell_{\gamma}, following the proof of Lemma 11 in [ghorbani2021linearized] with N_{\ell_{\gamma}} being replaced by N_{\ell_{\gamma}-1}, we can similarly show (G.3). The remaining part of Lemma G.3 is a direct result of Theorem 1 in [xiao2022precise].
Suppose \gamma = \ell_{\gamma}. Denote \overline{D}:= (n\lambda + 1) \cdot n^{-1}\Sigma_{\leq \ell_{\gamma}}^{-1} + n^{-1}\Psi_{\leq \ell_{\gamma}}^{\top} \Psi_{\leq \ell_{\gamma}}, then we have
We have the following bounds:
(i) \lambda_{\text{min}}\left( M\right) = \Omega_{d, \mathbb{P}}(n\lambda+1), \mathrm{tr}(M / n)=O_{d, \mathbb{P}}(n\lambda+1).
(ii) M_{>\ell_{\gamma}} = K_{>\ell_{\gamma}} + n\lambda \mathrm{I}_n = \Theta_{d, \mathbb{P}}(n\lambda + 1) \left(\mathrm{I}_{n} + \Delta_{1}\right).
(iii) When \ell_{\lambda}+1 \leq \ell_{\gamma}, for any i=N_{\ell_{\lambda}}+1, \cdots, \lfloor\min\{n, N_{\ell_{\lambda}+1}\}\rfloor,
we have
(iv) When \ell_{\lambda}<\ell_{\gamma}, we have
and \lambda_{\text{max}}(K_{>\ell_{\lambda}}) = O_{d, \mathbb{P}}\left({d^{\gamma-\ell_{\lambda}-1}}\right) = o_{d, \mathbb{P}}(n\lambda).
(v) Denote A = n^{-1}\Sigma_{\leq \tilde{\ell}}^{-1} + n^{-1}\Psi_{\leq \tilde{\ell}}^{\top} M_{>\tilde{\ell}}^{-1} \Psi_{\leq \tilde{\ell}}, then we have
(i) From Lemma G.1, we have
Further, from Lemma G.1, we have
(ii) It is a direct corollary of Lemma G.1.
(iii) Denote the SVD decomposition of \Psi_{\leq \ell_{\gamma}} as \Psi_{\leq \ell_{\gamma}} = n^{1/2} ~ O H V^{\top}, where O \in \mathbb{R}^{n \times n} and V \in \mathbb{R}^{N_{\ell_{\gamma}} \times N_{\ell_{\gamma}}} are orthogonal matrices, H = \left[ H_{\star}; 0 \right]^{\top} \in \mathbb{R}^{n \times N_{\ell_{\gamma}}}, and H_{\star} \in \mathbb{R}^{N_{\ell_{\gamma}} \times N_{\ell_{\gamma}}} is a diagonal matrix. Then, Lemma G.2 and Lemma G.3 imply that both \lambda_{\text{max}}\left(H_{\star}\right) and \lambda_{\text{min}}\left(H_{\star}\right) are of order \Theta_{d, \mathbb{P}}(1).
Recall the assumption that \ell_{\lambda}+1 \leq \ell_{\gamma}. Hence, for any i=N_{\ell_{\lambda}}+1, \cdots, \lfloor\min\{n, N_{\ell_{\lambda}+1}\}\rfloor, from Lemma G.1, we have
(iv) When \ell_{\lambda}<\ell_{\gamma}, we have 1 = o_d(n\lambda) and nd^{-\ell_{\lambda}-1}=\Omega_d(1). As in Case (i), we can show that \lambda_{\text{min}}(M_{>\ell_{\lambda}}) = \Omega_{d, \mathbb{P}}(n\lambda+1) = \Omega_{d, \mathbb{P}}(n\lambda). On the other hand, we have
Denote the SVD decomposition of \Psi_{\ell_{\lambda}+1, \ell_{\gamma}} as \Psi_{\ell_{\lambda}+1, \ell_{\gamma}} = n^{1/2} ~ O H V^{\top}, where O \in \mathbb{R}^{n \times n} and V \in \mathbb{R}^{(N_{\ell_{\gamma}}-N_{\ell_{\lambda}}) \times (N_{\ell_{\gamma}}-N_{\ell_{\lambda}})} are orthogonal matrices; H = \left[ H_{\star}; 0 \right]^{\top} \in \mathbb{R}^{n \times (N_{\ell_{\gamma}}-N_{\ell_{\lambda}})} and H_{\star} \in \mathbb{R}^{(N_{\ell_{\gamma}}-N_{\ell_{\lambda}}) \times (N_{\ell_{\gamma}}-N_{\ell_{\lambda}})} is a diagonal matrix. Then, Lemma G.2 and Lemma G.3 imply that both \lambda_{\text{max}}\left(H_{\star}\right) and \lambda_{\text{min}}\left(H_{\star}\right) are of order \Theta_{d, \mathbb{P}}(1). Therefore,
Combining all these, we have that both \lambda_{\text{max}}(M_{>\ell_{\lambda}}) and \lambda_{\text{min}}(M_{>\ell_{\lambda}}) are of order \Theta_{d, \mathbb{P}}(n\lambda). Finally, as in (?) and (?), we can show that \lambda_{\text{max}}(K_{>\ell_{\lambda}}) = O_{d, \mathbb{P}}(nd^{-\ell_{\lambda}-1}+ 1)=O_{d, \mathbb{P}}(nd^{-\ell_{\lambda}-1})=o_{d, \mathbb{P}}(n\lambda), finishing the proof of (iv).
(v) We have A = n^{-1}\Sigma_{\leq \tilde{\ell}}^{-1} + n^{-1}\Psi_{\leq \tilde{\ell}}^{\top} M_{>\tilde{\ell}}^{-1} \Psi_{\leq \tilde{\ell}}. On the one hand, from Lemma G.2, Lemma G.3, (ii), and (iv), we have
On the other hand, to obtain the lower bound of \lambda_{\text{min}} \left( A \right), we consider the following three cases: (a) \gamma >\ell_{\gamma}; (b) \gamma = \ell_{\gamma} and \ell_{\lambda} < \ell_{\gamma}; (b) \gamma = \ell_{\gamma} and \ell_{\lambda} \geq \ell_{\gamma}.
Case (a). When \gamma >\ell_{\gamma}, from Lemma G.2, (ii), and (iv), we have
Case (b). When \gamma = \ell_{\gamma} and \tilde{\ell}= \ell_{\lambda} < \ell_{\gamma}, from Lemma G.3 and (iv), we have
Case (c). When \gamma = \ell_{\gamma} and \ell_{\lambda} \geq \ell_{\gamma}, we have
and we get the desired results.
We have
(i) \left\| \Psi_{>\tilde{\ell}} \theta_{>\tilde{\ell}} \right\|_{2}^{2} = O_{d, \mathbb{P}}\left( n \| \theta_{>\tilde{\ell}} \|_{2}^{2}\right).
(ii) \| \theta_{>\tilde{\ell}} \|_{2}^{2} = \Theta_d \left( d^{-(\tilde{\ell}+1)s} \right).
(iii) \left\| \Sigma_{\leq \tilde{\ell}}^{-\tau^{\prime}} \theta_{\leq \tilde{\ell}} \right\|_{2}^{2} = \Theta_{d} \left( d^{(2\tau^{\prime}-\min\{s, 2\tau^{\prime}\})\tilde{\ell} } \right), \tau^{\prime} \geq 1.
(i) and (ii) are direct results of Lemma S8 and S10 in [zhang2024phase]. Now let’s prove (iii). We have
On the one hand, when 0 \leq s \leq 2\tau^{\prime}, (Proof) implies
On the other hand, when s > 2\tau^{\prime}, since \mu_m = O_d(1), m=0, 1, \cdots, \tilde{\ell}, (Proof) implies
and we obtain the desired results.
For k \in \{\ell_{\gamma}, \ell_{\lambda}\}, we have
and
Recall that we have M = \Psi_{\leq k} \Sigma_{\leq k} \Psi_{\leq k}^-{\top} + M_{>k}.
Applying the Sherman-Morrison-Woodbury formula with Z= \Psi_{\leq k} \Sigma_{\leq k}^{1/2} and A=M_{>k}, we have
Applying the Sherman-Morrison-Woodbury formula with Z= \Psi_{\leq k} M_{> k}^{-1/2} and A=\Sigma_{\leq k}^{-1}, we have
We have
and
with
Moreover, denote \tilde{N}_{\ell_{\gamma}} = \sum_{k=0}^{\ell_{\gamma}}\tilde{N}(d, k), then we have
While Theorem 6 in [mei2022generalization] is stated under the setting of random feature kernels, its proofs (mainly relying on Propositions 3 and 4 in [mei2022generalization]) do not rely on any property specific to this class. Consequently, the results remain valid for general kernels, and we adopt them in this broader setting.
Proof of (H.1) and (H.1). We first state the following conditions (i.e., Assumption 6 (a), Assumption 6 (b), and Assumption 7 (a) in [mei2022generalization] with the following substitution of notations: u(d) \to \sum_{k=0}^{2\ell_{\gamma} +1}\tilde{N}(d, k), N(d) \to n, M(d) \to \sum_{k=0}^{\ell_{\gamma}}\tilde{N}(d, k), and \{\lambda_{d, j}^2\} \to \{\tilde{\lambda}_{k, j}\}):
For any integer q \geq 1, there exists C=C(q, \ell), such that for any k \leq 2\ell + 1 and any j \leq \tilde{N}(d, k), we have
There exists \delta_1>0, such that
There exists \delta_2>0, such that
Now let’s verify these conditions. Since \|\tilde{\psi}_{k, j}\|_{L^{2}}=1, Assumption 9 implies (?). Assumption 8 implies that \sum_{j \leq \tilde{N}(d, k)} \tilde{\lambda}_{k, j} =\Theta(1) for any k\leq 2\ell_{\gamma}+2. Together with Assumption 7, it further implies that there exist positive constants \delta_1 and \delta_2 such that
and
Therefore, Theorem 6 in [mei2022generalization] ensures that both (H.1) and (H.1) hold.
Proof of (H.1). Assumption 7 and 8 imply that there exist \delta_3, \delta_4>0, such that
and
Similar as above, from Theorem 6 in [mei2022generalization] with the substitution
we have that (H.1) holds.
We have
(i) \left\| \tilde{\Psi}_{>\tilde{\ell}} \theta_{>\tilde{\ell}} \right\|_{2}^{2} = O_{d, \mathbb{P}}\left( n \| \theta_{>\tilde{\ell}} \|_{2}^{2}\right).
(ii) \| \theta_{>\tilde{\ell}} \|_{2}^{2} = \Theta_d \left( d^{-(\tilde{\ell}+1)s} \right).
(iii) \left\| \tilde{\Sigma}_{\leq \tilde{\ell}}^{-1} \theta_{\leq \tilde{\ell}} \right\|_{2}^{2} = \Theta_{d} \left( d^{(2-\min\{s, 2\})\tilde{\ell} } \right).
Adopt the notations in (?). We divide the proof into four steps.
Step (I). We first show that results in Appendix ? can be generalized to the cases where kernels are defined on general domains.
Recall that \gamma > \ell_{\gamma}. We have:
Since the proof of Lemma G.6 relies on Lemma G.1 and Lemma G.2, it can be established in a similar manner by replacing them with Lemma H.1.
Lemma G.8 still holds with each symbol augmented with a tilde.
Step (II). We next show that the results for the variance term in Appendix ? can also be extended to the case where kernels are defined on general domains.
Since the proof of Theorem D.1 depends on the results established in Appendix ?, Step (I) directly implies that
Step (III). We then bound the bias term by modifying the proof in Appendix ?.
From (?) we have
In the proof of Theorem E.1, since results for B_2, B_3, B_4 and B_5 depend on the results established in Appendix ?, Step (I) directly implies that (see also (?)):
We then bound the term \|B_1\|_2^2. Notice that for kernel ridge regression, we have
Moreover, from Step (I), the generalization of Lemma G.8 and Lemma G.6 (v) imply that
Therefore, for B_1, we have
Step (IV). Finally we combine the estimates from Step (II) (variance) and Step (III) (bias decomposition) to obtain the learning curve.
From Lemma F.5 we have b_3 \leq v_2. Therefore:
Upper bound:
For lower bound, we analyze three possible cases:
Suppose b_3 < b_2(u, 1) and b_1 < b_2(u, 1). Then
Suppose b_3 < b_2(u, 1) and b_1 \geq b_2(u, 1). Then
Suppose b_3 \geq b_2(u, 1). Then we further have \max\{b_2(u, 1), b_3\} \leq v_2, and hence
In sum, we have
and we obtain the desired results.
Proof of (i). Suppose 0 \leq s \leq \Gamma(\gamma).
Recall that we have p := \lfloor \gamma/(s+1) \rfloor \leq \ell_{\gamma} := \lfloor \gamma \rfloor. The proof is split into the following three steps:
Step I. We first consider the case where \gamma > 0 is an integer. Then (16) and 0 \leq s \leq \Gamma(\gamma) imply that s=0. Corollary F.1 implies that
where the last equation follows from p \leq \gamma and s=0.
Step II. We then consider the case where \gamma > 0 is not an integer and show p=\ell_{\gamma} under the following three cases:
When \gamma \leq 0.5, we have p=\ell_{\gamma}=0;
When \gamma \in (\ell_{\gamma}, \ell_{\gamma}+0.5] for \ell_{\gamma}=1, 2, \cdots, we have
When \gamma \in (\ell_{\gamma}+0.5, \ell_{\gamma}+1), we have
Furthermore, Proposition F.4 implies that p \leq u^{\prime}(\tau) < \gamma. Therefore, for any u \in [u^{\prime}(\tau), \infty], we have
Step III. Now we are ready to determine the convergence rate of the excess risk when \gamma > 0 is not an integer. We first recall the definitions of certain notations, e.g., b_i = b_i(u) and v_i = v_i(u) for i=1,2, in Appendix ?.
From Theorem 3.1, we have
Furthermore, Proposition F.6 implies that, for any u \in [u^{\prime}(\tau), \infty], we have
We divide the remaining proof in two parts:
When u \geq \gamma, we have
Therefore, (?) implies
Notice that the above rate is the same as the rate given in (13) of [zhang2024phase]. Following the same discussion under Corollary 1 in [zhang2024phase], we have
(For completeness, we provide an alternative proof of (?) in Remark I.1.)
When u^{\prime}(\tau) \leq u < \gamma, from (?) we have
When \gamma \in (\ell_{\gamma}, \ell_{\gamma}+0.5], from (?) we have u<\ell_{\gamma}+0.5=p+0.5, hence
When \gamma \in (\ell_{\gamma}+0.5, \ell_{\gamma}+1), we have
Therefore, for any u \in [u^{\prime}(\tau), \infty], from (?) we have
Proof of (ii). Suppose s > \Gamma(\gamma). When u=\gamma, we have \tilde{\ell}=\ell_{\gamma}, hence Theorem 3.1 implies
Notice that the above rate is the same as the one given in (13) of [zhang2024phase]. Therefore, following the same discussion under Corollary 1 in [zhang2024phase], we have
(For completeness, we provide an alternative proof of (?) in Remark I.2.)
Recall that Proposition F.4 implies \gamma > u^{\prime}. Notice that, for any non-integer u, Theorem 3.1 implies that the convergence rate of the excess risk E_{x, \epsilon} \left[ \left(\hat{f}_{\lambda}(x) - f_{\star}(x) \right)^{2} \right] is continuous with respect to u, hence there exists \tilde{u} \in (u^{\prime}, \gamma), such that for \tilde{\lambda}=d^{-\tilde{u}}, we have
Here we provide an alternative proof for (?). Notice that
where the first case follows from 0< \gamma - \ell_{\gamma} \leq 0.5 and the second case follows from the definition of \Gamma(\cdot) and the assumption that s\geq \Gamma(\gamma). Recall that in Step II we have shown that p = \ell_{\gamma}, therefore we have
Here we provide an alternative proof for (?). We consider four different cases given by (16).
Case 1: \gamma \in (0, 0.5]. In this case we have \Gamma(\gamma)=\infty, hence it is impossible to let s>\Gamma(\gamma).
Case 2: \gamma \in (\ell_{\gamma}, \ell_{\gamma}+0.5] ~\text{ and }~ \ell_{\gamma} \geq 1. In this case we have \gamma / (s+1) < \gamma / (\Gamma(\gamma)+1)=\ell_{\gamma}, and hence p<\ell_{\gamma}. Therefore, we have
where the first inequality follows from 0 < \gamma - \ell_{\gamma} \le \tfrac{1}{2}.
Therefore, we have
Furthermore, we have
where the first case uses s>\Gamma(\gamma) and (16), and the second case uses \gamma < (p+1)(s+1) by the definition of p.
Together with p<\ell_\gamma, we have
Case 3: \gamma \in (\ell_{\gamma}+0.5, \ell_{\gamma}+1). In this case we have
On one hand, when p=\ell_{\gamma}, we have
On the other hand, when p<\ell_{\gamma}, we have
where the last inequality follows from \gamma<(p+1)(s+1). Therefore, we have
Case 4: \gamma =1, 2, \cdots. In this case we have s>\Gamma(\gamma)=0. Since \gamma / (s+1) < \gamma = \ell_{\gamma}, we further have p<\ell_{\gamma}. Therefore, d^{\ell_{\gamma} - \gamma} is of order \Theta_d(1) while d^{p-\gamma} + d^{-(p+1)s} is of order o_d(1). Hence we have
Consider the sequence model in (17). Suppose \lambda_j’s are known and consider the parametrization f_j = \lambda_j^{1/2} \beta_j, j=1, 2, \cdots. Then, applying ridge regression with regularization parameter \lambda or gradient flow with early-stopping time t = \lambda^{-1} yields estimators of the form
where \varphi_\lambda is the regularization function \varphi_\lambda^{\mathtt{KRR}} or \varphi_\lambda^{\mathtt{GF}} defined in Section 2.3.1.
Ridge regression. Considering the loss L(\hat{\beta})=\frac{1}{2}\sum_{j=1}^{\infty} (z_j - \lambda_j^{1/2} \hat{\beta}_j)^2 + \frac{\lambda}{2} \sum_{j=1}^{\infty} \hat{\beta}_j^2. Minimizing the above loss with respect to each \hat{\beta}_j yields the closed-form solution \hat{\beta}_j = \lambda_j^{1/2} (\lambda_j + \lambda)^{-1} z_j = \lambda_j^{1/2} \varphi_\lambda^{\mathtt{KRR}}(\lambda_j) z_j. Consequently, the estimator for \theta_j is given by:
Gradient flow. Considering the loss L(\hat{\beta})=\frac{1}{2}\sum_{j=1}^{\infty} (z_j - \lambda_j^{1/2} \hat{\beta}_j)^2. The gradient flow dynamics, initialized at \hat{\beta}_j(0) = 0, are governed by the differential equation:
and the solution to this equation is \hat{\beta}_j = \lambda_j^{-1/2} (1+e^{-\lambda_j t}) z_j= \lambda_j^{1/2} \varphi_\lambda^{\mathtt{GF}}(\lambda_j) z_j. Therefore, the corresponding estimator for \theta_j at time t = \lambda^{-1} is:
and we obtain the desired results.
Denote
Then, when \lambda > 0, Lemma D.14 in [lu2024saturation] implies that
Therefore, (19) implies
Let \tilde{\mathscr{K}}_{\alpha, r} be defined as (?). Then it is a continuous and positive-definite kernel on \mathcal{X} \times \mathcal{X} with respect to \rho_{\mathcal{X}}, where \mathcal{X} = [0, \infty)^d, and \rho_{\mathcal{X}} is the product measure with each coordinate x_{(i)} following an independent Gamma distribution \Gamma(\alpha_i + 1, 1). Moreover, it admits a Mercer decomposition given in (22), with eigenvalues and eigenfunctions satisfying Assumption 8 with \tilde{\mathfrak{C}} = 4^r.
It is obvious that \tilde{\mathscr{K}}_{\alpha, r} is a continuous and positive-definite kernel.
We then show that
and provide explicit expressions of \tilde{\lambda}_{k, j}’s and \tilde{\psi}_{k, j}’s. For any c>0, let \{L_{k}^{(c)}(\cdot)\}_{k = 0}^{\infty} be the generalized Laguerre polynomials (see, e.g., Chapter 6.2 in [andrews1999special])
and \tilde{L}_{k}^{(c)}:=\sqrt{k! \Gamma(c+1) / \Gamma(k+c+1)}L_{k}^{c}(\cdot), k \geq 0, then we have (see, e.g., Eq. (6.2.3) in [andrews1999special]):
Furthermore, it is known that the generalized Laguerre polynomials satisfy the Hardy–Hille formula (see, e.g., Eq. (2.13) in [chatterjea1966operational]), that is,
Combining (?) and (?), we have
implying that \tilde{\lambda}_{k, j} = (r/d)^{k} and \tilde{N}(d, k) = C_{k+d-1}^{d-1}. Therefore, we have
Moreover, for any k = 0, 1, \cdots, 2\ell_{\gamma} +2, we have
Suppose A, B are semi-positive definite matrices, A+B is a positive definite matrix. Then we have
Denote M:=A+B. Recall that \psi_\lambda(M) = \mathrm{I} - M \varphi_\lambda(M). Then
Let A_1 and A_2 be symmetric matrices on \mathbb{R}^{n \times n}, with their eigenvalues ordered in descending order. Then, for any indices i and j, we have:
For Hermitian n \times n positive semi-definite complex matrices A and B, where now the eigenvalues are sorted decreasingly (a_1 \geq a_2 \geq \cdots \geq a_n \geq 0 and b_1 \geq b_2 \geq \cdots \geq b_n \geq 0, respectively), we have
Let \{A_i\}_{i=1}^n be a collection of n independent, zero-mean matrices in \mathbb{R}^{d_1 \times d_2}, such that \|A_i\|_2 \leq b almost surely. Let
Then, for any \delta > 0, the following tail bound holds:
Suppose k < n, A \in \mathbb{R}^{n \times n} is an invertible matrix, and Z \in \mathbb{R}^{n \times k} is such that ZZ^{\top} + A is invertible. Then we have
The text on this page is from arXiv:2604.23212v1 by Weihao Lu, Qian Lin, Yingcun Xia and Dongming Huang, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2604/2604.23212.jsonl.