Moving Least Squares without Quasi-Uniformity: A Stochastic Approach

The proof follows immediately from Lemma 1 and Lemma 2 below.

The key is to express the shape functions through the unnormalized Gram matrix \mathscr{G}_n of (?), which—unlike the normalized matrix \mathscr{A}_n=\tfrac{1}{N_{B(x,h_n)}}\mathscr{G}_n, depends smoothly on x. By (?), \eta(x)=\mathscr{G}_n(x)^{-1}\boldsymbol{p}(x), hence

The sample count N_{B(x,h_n)} has cancelled and does not appear in (Proof). We now show the smoothness of \mathscr{G}_n. By (?),

Each summand is the product of the \mathcal{C}^k weight \theta_h(x_t-x) (which vanishes to order k at the boundary of its support) with polynomials, hence is \mathcal{C}^k in x; a sample x_t entering or leaving the support B(x,sh_n) does so with \theta_h\to 0 smoothly. Thus \mathscr{G}_{n\,ij}\in\mathcal{C}^k(\Omega). In contrast to \mathscr{A}_n, the Gram matrix carries no count normalization, so it does not inherit the discontinuity of x\mapsto N_{B(x,h_n)}.

Now, Fix x_0\in\Omega and work on the event \mathscr{E}_I throughout. By Lemma 7, with conditional probability at least 1-\tfrac{2A^2}{n^{\phi}} we have \lambda_{\min}(\mathscr{A}_n(x_0))\ge C_\lambda; on this event, since \mathscr{G}_n(x_0)=N_{B(x_0,sh_n)}\,\mathscr{A}_n(x_0) and, by \mathscr{E}_I, N_{B(x_0,sh_n)}\ge\gamma_1\log n,

To propagate this to a ball, we bound the Lipschitz constant of the entries of \mathscr{G}_n. Any sample contributing to \mathscr{G}_n(x) for some x\in B(x_0,\delta_0) with \delta_0\le sh_n lies within 2sh_n of x_0; the number of such samples is at most the count in B(x_0,2sh_n), which by Lemma 3 (applied with radius 2sh_n) is at most \gamma_2'\log n for a constant \gamma_2'; on \mathscr{E}_I this holds together with the lower bound (?). Differentiating an entry brings one factor h_n^{-1} through \theta_h, so there is a constant C_{\mathrm{Lip}}, independent of n, with

With Weyl’s inequality and the norm equivalence \|B\|_{\mathrm{op}}\le C_{\mathrm{norm}}\|B\|_{F,1},

Set

The two factors of \log n cancel, leaving \delta_0\asymp h_n.

For every x\in B(x_0,\delta_0), the Lipschitz bound together with \|x_0-x\|\le\delta_0 and the definition of \delta_0 gives

where the last inequality uses (?). Hence

so \mathscr{G}_n(x) is positive definite throughout B(x_0,\delta_0).

Therefore, on B(x_0,\delta_0) the matrix \mathscr{G}_n(x) is \mathcal{C}^k and invertible; by Cramer’s rule its inverse \mathscr{G}_n(x)^{-1} is \mathcal{C}^k there (entries are \mathcal{C}^k cofactors divided by \det\mathscr{G}_n(x), which is bounded away from 0). Hence \eta(x)=\mathscr{G}_n(x)^{-1}\boldsymbol{p}(x)\in\mathcal{C}^k, and by (Proof) with \theta_h\in\mathcal{C}^k, each shape function a_i^*(x)\in\mathcal{C}^k(B(x_0,\delta_0)). Therefore

Let N_{D_n} = \sum_{i=1}^n \mathbf{1}_{\{x_i \in D_n\}}. Each indicator variable \mathbf{1}_{\{x_i \in D_n\}} is Bernoulli with success probability p_n = P(x_i \in D_n). Then N_{D_n} \sim \text{Bin}(n, p_n), and:

Since D_n is a ball of radius R_n \propto \left(\frac{\log n}{n}\right)^{1/d}, its volume is

Because p(x) is bounded below by a constant \underline{c} > 0 on its support, we obtain:

for some constant c_1 > 0. Likewise, since p(x) \leq \bar{c}, we also get:

We now apply the Chernoff bound: If X \sim \text{Bin}(n, p) with \mu = \mathbb{E}X, then for any 0 < \delta < 1,

Applying this to N_{D_n} with \mu = \mathbb{E} N_{D_n}, we obtain:

where c is a positive constant dependents on \delta. Thus,

Denote \gamma_1=(1-\delta)c_1 and \gamma_2=(1+\delta)c_2, and use the inequalities in (Proof),(Proof) we obtain that

which implies that

Let X_n=\left\{x_i\right\}_{i=1}^n\subset\mathbb{R}^d be a set of scattered data points.

In the following, we work under the assumption that the event \mathscr{E}_I defined in (?) holds; under this event, by (Proof), the expected number of samples in the index set I_{B(\hat{x}, s h_n)} is contained in [\gamma_1 \log n,\gamma_2 \log n] with probability at least 1 - \frac{2}{n^c}.

In [levin1998approximation] it was shown that the MLS approximation is given by

where the coefficients a_i^*(x) are called shape functions. These functions are obtained by minimizing a quadratic form, a procedure rooted in the foundational work of Backus and Gilbert [backus1968resolving, backus1970uniqueness] regarding optimal local averaging kernels. Specifically, the shape functions are defined as:

under the constraints

and \theta_h(x_i-x) is a weight function satisfies the properties in (?). Let A denote the number of multi-indices \alpha = (\alpha_1, \dots, \alpha_d) with nonnegative entries such that |\alpha|\leq k. For a multi-index \alpha and a vector v \in \mathbb{R}^d, we define v^\alpha = \prod_{j=1}^d v_j^{\alpha_j}, where |\alpha| = \sum_{j=1}^d \alpha_j denotes the order of the multi-index. It is shown in [mirzaei2015analysis] that the shape functions can be expressed in the following form:

where \eta_\alpha(x) are the components of \eta(x)\in\mathbb{R}^A. The polynomial–reproduction property of the shape functions determines \eta(x) as the solution of the positive definite system

where \boldsymbol{p}(x)\in \mathbb{R}^A is the vector whose components are p_\alpha(x),

and \mathscr{G}_n(x)\in \mathbb{R}^{A\times A} is the (unnormalized) Gram matrix with entries

The normalized moment matrix \mathscr{A}_n(x) analyzed below—the empirical weighted covariance matrix, which converges to the theoretical weighted covariance matrix as n\to\infty—is obtained by averaging \mathscr{G}_n over the active samples,

Consequently the reproduction system (?) is equivalent to

To illustrate the approximation of a function by the stochastic MLS operator, we follow Levin’s approach [levin1998approximation]. Given an arbitrary \hat{x}\in\Omega, we define the MLS error as

By adding and subtracting a polynomial p(x) of degree \leq k and centered around \hat{x}, we get

Denote by p_f the taylor polynomial of f(x) around \hat{x}. We have:

where the seminorm, |f|_{\mathcal{C}^k\left(B\left(\hat{x},s h_n\right)\right)}, is defined by

with \alpha=\left(\alpha_1, \ldots, \alpha_d\right) denoting a multi-index of order |\alpha|=k, and

Using p_f as the polynomial in Equation (?), (?) we get

Since the focus is on measuring the error in the derivatives, we obtain

Note that in the current case, \partial^\alpha is scalar-valued as f: \mathbb{R}^d \rightarrow \mathbb{R}. Then, for a fixed but arbitrary \hat{x} \in \Omega we get

Since p_f is the Taylor expansion of f(x) at \hat{x} and \|\hat{x}-x\| \leq sh_n we get that

Given the event \mathscr{E}_I hold, we show in Lemma 9 that for all x \in B(\hat{x},s h_n),

where C'_{\alpha} is a positive constant. Therefor, using (?) to bound the probability of the event \mathscr{E}_I, we have,

Hence, exists a constant c_{\mu} such that

Substituting the bounds from Equations (?) and (?) into Equation (?), and using the fact that \max_{x \in B(\hat{x},h_n)} |f(x) - p_f(x)| is bounded by h_n^k up to a positive constant (Taylor remainder estimate, (?)), we evaluate at \hat{x}, and we obtain that exits a positive constant \mathcal{K} such that

That is,

as desired.

Let B\left(\hat{x}, h_n\right)=\left\{x \in \mathbb{R}^d:\|x-\hat{x}\| \leq h_n\right\}. Denote I_{B\left(\hat{x}, h_n\right)}=\left\{i:X_i\in B(\hat x, h_n)\right\} and N_{B\left(\hat{x}, h_n\right)}=\#I_{B\left(\hat{x}, h_n\right)} is the number of elements in this ball. For \Pi_{k-1}^d we consider the basis

which is the monomial bases centered around the point \hat x and scaled by the fill distance. This choice of scaled monomial basis is made for convenience. The MLS formulation, and in particular the matrix \mathscr{A}_n and its derivatives, are invariant under a change of basis in \Pi_{k-1}^d. Consequently, the estimates derived below do not depend on the specific choice of polynomial basis and hold for any basis of \Pi_{k-1}^d. Thus, the MLS function can be written as

where A denotes the collection of all d-tuples \alpha of nonnegative integers such that |\alpha| \leq k-1. The expression \hat{b}_{n\alpha}(x) is the \alpha-coordinate of the vector of size A which is given by

where \mathscr{X}_n(x) and \mathscr{A}_n(x) are defined as follows: Using the basis defined in Equation (?), we can construct the normal equations accordingly (the full formulation of the least-squares problem and the solution of the resulting normal equations are given in the ?). The n\times A design matrix, \tilde{\mathscr{X}}_n(x), takes the form

where \theta_i=\theta_h(x_i-x) and p_{\alpha_j}; \,\,0\leq j\leq A is according to (?). From this, the normal equations are

which leads to the solution

Next, we normalize the design matrix by rescaling it:

i.e.,

Note, as stated in the Lemma, x is fixed, hence, once we fix x, the magnitude N_{B\left(x, h_n\right)} depends only on n. Using this notation, let \mathscr{A}_n(x) be the A \times A matrix defined by

Therefore, each matrix element is given by

With these notations, we obtain the terms \mathscr{X}_n(x), \mathscr{A}_n(x) and \hat{b}_{n }(x). Now, note that \mathscr{A}_{n} represents the empirical weighted covariance matrix of the random variable (i.e., the sample points x_1,...,x_n) expressed in the polynomial basis. Define the normalized weight function by

using change of variables with \left[t = \frac{u-x}{h_n}, u - x = t h_n, du = h_n dt\right] we obtain,

where t_i = \frac{x_i - x}{h_n}. Therefore,

so that w(x) is a probabilty density function over the ball B(0,sh). Hence, the normalized sample covariance matrix is given by

Hence,

Define the A\times A matrix \mathscr{A}^w by it’s entries

The matrix \mathscr{A}^w is symmetric and positive definite; consequently, its determinant is strictly positive, \operatorname{det} \mathscr{A}^w>0. Conceptually, \mathscr{A}^w is a weighted covariance matrix, and the matrices \mathscr{A}(x) and \mathscr{A}^w_n(x) both serve as covariance matrices of the basis functions, with \mathscr{A}^w_n(x) being a scaled version of \mathscr{A}(x) that approximates the theoretical weighted covariance matrix \mathscr{A}^w. The primary difference lies in the normalization factors, N_{B\left(x, h_n\right)} and \frac{1}{\int_{t\in B(0, s)}\Phi(t - t_i) dt}, making them equivalent up to a multiplicative constant.

By the Scalar Bernstein inequality, for every \varepsilon>0,

where c' is a positive constant. So if we denote C^{\Phi}=\int_{t\in B(0, s)}\Phi(t) dt, then by Equation (?)

Now, by Lemma 6 we obtain that for all |\alpha| \leq k, there exists a constant C_\alpha such that

Since \Phi(x) is a compactly supported and \mathcal{C}^k function, all of its derivatives up to order k are continuous and bounded (see Lemma 6). As only w carries the x-dependence, the bound in (?) applies verbatim to \partial^\alpha \mathscr{A}_n, whose mean is controlled by (?).

Combining Equations (?) and (?) together implies that

By choosing \varepsilon such that c'\varepsilon^2=\tilde{c}, we can simplify (?) to:

where \tilde{C}_\alpha = C_\alpha + \varepsilon and \tilde{c} is a positive constant.

Differentiate (?) with respect to x

Since the denominator is independent of x, differentiation only affects the numerator. Using the chain rule,

Thus,

Since \Phi and its derivatives are bounded, there exists a constant C_\alpha^{\prime} such that

Therefore,

The denominator is independent of x and is just a normalization factor ensuring w(x_i-x) integrates to 1 . Then if we denote C_\alpha=\frac{C'_\alpha}{\int_{t\in B(0, s)}\Phi(x_i/h_n - u/h_n) du} we obtain

Let \pi=\left(\pi_\alpha\right)_{\alpha \in A} be a unit vector in \mathbb{R}^{|A|} such that \sum_{\alpha\in A} \pi_\alpha^2=1. For z\in\mathbb{R}^d, we define the polynomial:

Now define the empirical quadratic form:

From (?) we get

Let us define the rescaled coordinates:

Substituting t_i=\frac{x_i-x}{h_n}, and using \theta_h\left(x_i-x\right)=\Phi\left(t_i\right), we get:

where P(t)=\sum_\alpha \pi_\alpha t^\alpha. Given the event \mathscr{E}_I, the number of samples \left\{t_i\right\}_{i \in I_{B\left(x, h_n\right)}} inside the rescaled ball B(0, s) goes to infinity as n \rightarrow \infty in probability. We now show that \mathbb{E}\left[\mathcal{Q}_n(\pi)\right] is positive.

By applying the change of variables t=\frac{u-x}{h_n}, which means d u=h_n^d d t, we obtain:

Recall now that \int_{B(0,s)} \underline{c} d \mu \leq \int_{B(0,s)} d p \leq\int_{B(0,s)} \bar{c} d \mu, where p is a density function satisfies Definition 4, hence,

where, \underline{\beta}= \frac{\underline{c}}{\bar{c}\cdot \text{Vol}B(0,s)} and \bar{\beta}=\frac{\bar{c}}{\underline{c}\text{Vol}B(0,s)}. Denote,

Thus,

Since the integrand \Phi(t) \cdot P(t)^2 is non-negative, and P(t) is not identically zero on B(0,s), the integral \mathcal{Q}(\pi) is strictly positive for all \|\pi\|=1, i.e., \mathcal{Q}(\pi)>0. Moreover, \mathcal{Q}(\pi) is continuous in \pi, and is defined on a compact support, therefore, by the Weierstrass Theorem \mathcal{Q}(\pi) is bounded and attains its minimum. Denote,

In particular, for every \pi, \mathcal{Q}(\pi)>0 so we also have \tau>0. We fix from here on any constant

Consider the deterministic (conditional) covariance matrix

so that for every unit vector \pi,

where the first inequality is (?). Because \mathscr{A}(x) is deterministic, taking the infimum over \|\pi\|=1 is legitimate and yields

Similarly to Equation (?), by the scalar Bernstein inequality, for every \varepsilon>0 and every \alpha,\beta,

For an A\times A matrix B, \|B\|_{\mathrm{op}}\le\|B\|_F\le A\max_{\alpha,\beta}|B_{\alpha\beta}|. A union bound over the (finitely many, n-independent) A^2 entries therefore gives, from (?),

Note that c' is the same constant as in (?): the constant is depending only on \Phi and the support radius s, and the conditioning on \mathscr{E}_I fixes the number of active samples N_{B(\hat{x},sh_n)}=\Theta(\log n), so the Bernstein–Hoeffding exponent has the same dependence c'\varepsilon^2\log n in both lemmas.

Choose \varepsilon=\dfrac{\underline{\beta}\,\tau-C_\lambda}{A} (so A\varepsilon=\underline{\beta}\,\tau-C_\lambda>0) and set \phi:=c'\varepsilon^2=\dfrac{c'\left(\underline{\beta}\,\tau-C_\lambda\right)^2}{A^2}>0. By Weyl’s eigenvalue perturbation inequality for Hermitian matrices and (?), we obtain

Hence

which is the assertion (with the n-independent constant 2A^2 in place of 2). In particular \mathscr{A}_n(x) is positive definite and full rank for all sufficiently large n.

Applying the multivariate Leibniz rule to the identity

we obtain the following recursive formula for the derivatives of the inverse matrix (see, e.g., [hormander1983analysis]), when we differentiating both sides with respect to a multi-index \alpha_0 and evaluating at a point x \in \Omega,

We now prove the desired result by induction on \left|\alpha_0\right|. Starting with the base case: \left(\left|\alpha_0\right|=1\right). Let \alpha_0=e_j, the unit multi-index in the j th direction. Then,

From Lemma 5 we have,

and from Lemma 7, we know that given the event \mathscr{E}_I,

Using the sub multiplicative property of the matrix norm, \|AB\|\leq\|A\|\|B\|, and the bounds from (?) and (?), we have:

By the union bound, the probability that both \left\|\mathscr{A}_n^{-1}(x)\right\| \leq 1 / C_\lambda and \left\|\partial^{e_j} \mathscr{A}_n(x)\right\| \leq \tilde{C}_\alpha h_n^{-1} hold is at least 1-\frac{2}{n^{\bar{c}}}-\frac{2}{n^\phi}. Thus, we conclude:

where C_{e_j}=\tilde{C}_\alpha / C_\lambda^2. This establishes the base case of the induction. Next, the inductive step assumes that for all multi-indices \beta<\alpha_0, we have for all x \in \Omega

Now we prove the inductive step. From the derivative identity we obtain:

Using the induction hypothesis (?) together with Equations (?),(?), and the bounds from Lemma 5 for \alpha_0-\beta, we obtain that,

where C_{\alpha_0}=\frac{1}{C_\lambda} \sum_{\beta<\alpha_0}\binom{\alpha_0}{\beta} \tilde{C}_{\alpha_0-\beta} C_\beta. This completes the inductive argument, as desired. By taking a positive constant s_1<\min\{\tilde{c},\phi\}, the constant factor in the tail bound can be absorbed for n>1. Thus,

which completes the proof.

Recall the stochastic MLS shape functions (?) are given by

We start with bounding the derivatives of \eta(x)=\dfrac{1}{N_{B(\hat{x}, s h_n)}}\,\mathscr{A}_n(x)^{-1} \boldsymbol{p}(x) (?), where the count N_{B(\hat{x}, s h_n)} does not depend on the differentiation variable x (it is the count at the fixed center \hat{x}), and is therefore carried as a constant scalar under \partial^\alpha.

Now, by the definition of the scaled polynomial basis (?), we have:

where e_\zeta is the canonical basis vector in multi-index ordering (i.e., the standard basis vector with a 1 in the position corresponding to multi-index \zeta, and zeros elsewhere). By Lemma 8, with probability of at least 1-\frac{1}{n^{s_1 }}, we have

where C'_\gamma is a positive constant independent of x and h_n. Hence, with the same probability, for a constant C_{\alpha,\zeta}:

where the vector C_\alpha\in\mathbb{R}^{|A|} is a bound for \sum_{\zeta \leqslant \alpha}\binom{\alpha}{\zeta} C_{\alpha, \zeta} e_\zeta. Taking the derivatives of both sides in Equation (Proof) and using the product rule, we have,

By Lemma 6, the weight functions satisfy:

and since x_i \in B\left(x, s h_n\right), we have \left|\left(x_i-x\right)^\beta\right| \leq\left(s h_n\right)^{|\beta|}. Applying the latter in addition to the previous bounds, we get that with probability of at least 1-\frac{1}{n^{s_1 }},

Summing over the active indices and using that the number of nonzero terms is exactly \#I_{B(\hat{x}, s h_n)}=N_{B(\hat{x}, s h_n)}, the factor N_{B(\hat{x}, s h_n)} cancels the 1/N_{B(\hat{x}, s h_n)} in the per–term bound:

Hence, on the event \mathscr{E}_I (which holds with probability at least 1-\tfrac{2}{n^c} by Lemma 3), for all x\in\Omega,

where C'_\alpha is a positive constant independent of x and h_n, as desired.

The proof of Theorem 3 established that, on the event \mathscr{E}_I and with conditional probability at least 1 - \tfrac{2A^2}{n^{\phi}}, the Gram matrix \mathscr{G}_n(x) is \mathcal{C}^k and positive definite for all x \in B(x_0,\delta_0),

with \det\mathscr{G}_n(x) bounded away from 0. By Cramer’s rule the entries of \mathscr{G}_n(x)^{-1} are then in \mathcal{C}^k on B(x_0,\delta_0), so the coefficients \eta(x) = \mathscr{G}_n(x)^{-1}\boldsymbol{p}(x) are in \mathcal{C}^k; together with the smoothness of the weight \theta_h and (Proof), each shape function a^*_i(x) is in \mathcal{C}^k\!\big(B(x_0,\delta_0)\big).

For the second claim, fix a multi-index \alpha with |\alpha| \le k. Since a^*_i \in \mathcal{C}^k\!\big(B(x_0,\delta_0)\big), the partial derivative \partial^\alpha a^*_i is continuous on B(x_0,\delta_0). The closed ball {B(x_0,\delta_0)} is compact, and a continuous function on a compact set is uniformly continuous; hence \partial^\alpha a^*_i is uniformly continuous on B(x_0,\delta_0), with the same conditional probability 1 - \tfrac{2A^2}{n^{\phi}}.