From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian

Consider the vector parameter \boldsymbol{\beta}_n=(\beta_{n,1},\dots,\beta_{n,p_n}) of some problem. To the best of our a priori knowledge the \beta_{n,i}s are exchangeable and |\beta_{n,i}|\le 1. I.e., their order has no meaning and all permutations are a priori equally likely. Any subjective prior over the parameter set have some properties. For example, if S_p\subset {1,\dots,p} and |S_p|(1-|S_p|)\to\ensuremath{\infty}\xspace (where |S| is the number of entries in the set S), then ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

1|S_p|_jS_p_n,j - 1p-|S_p|_jS_p_n,j

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

1|S_p|_jS_p_n,j - 1p-|S_p|_jS_p_n,j

is of order \max\{ \sqrt{1/|S_p|},\sqrt{1/(p-|S_p|)}\bigr\}. This is a very strong claim, which “most” sequences do not follow. The persistency of a hones prior claimed above is valid only on the rare sequences that obeys such conditions.

We consider the following Neyman-Scott type model as a concrete example. Our sample is all the patients from the n hospitals in the country during a year. The data do not include the identity of the hospital to keep privacy, but since we consider a meaningful hospital effect, we know which patients belong to same hospital. There are only a few patients per hospital, and for simplicity we assume that we have only two (or the first two patients during the year). We consider a mixed model. Let (g_i,h_i), i=1,\dots,n be some unobserved values, The model assumes: ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

X_i &= g_i + _i& &
Y_i &= h_i+X_i+_i
X_i &= g_i + _i& &
Y_i &= h_i+X_i+_i,

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

X_i &= g_i + _i& &
Y_i &= h_i+X_i+_i
X_i &= g_i + _i& &
Y_i &= h_i+X_i+_i,

where (\ensuremath{\varepsilon}\xspace_i,\eta_i) and (\tilde \ensuremath{\varepsilon}\xspace_i, \tilde \eta_i) are i.i.d. normal independent of (g_i,h_i). The regression parameter \ensuremath{\vartheta}\xspace is the only one of interest. We assume that a priori there is no relation between \ensuremath{\vartheta}\xspace and the gs and hs. Note that ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

Y_i-Y_i&= ( X_i - X_i)+ _i-_i.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

Y_i-Y_i&= ( X_i - X_i)+ _i-_i.

Thus, it is very easy to estimate \ensuremath{\vartheta}\xspace. The conservative frequentist can just consider the regression of \tilde Y_{i}-Y_{i} on \tilde X_{i} - X_{i} which are free from the nuisance parameters g_1,\dots,g_n and h_1,\dots,h_n. The Bayesian would do something similar, which depends on her prior on g_1,\dots,g_n and h_1,\dots,h_n, but asymptotically similar to the frequentist approach. Assuming that the (g_i,h_i) are i.i.d. observations is quite reasonable and safe (cf. [efron2019bayes, ritov2024no]). If she assumes further that they are jointly normal, the observations (X_i,Y_i,\tilde X_i,\tilde Y_i) are jointly normal, with observations consisting of the two different means, and the ten entries of the covariance matrix of the four variables, while the parameters are \ensuremath{\vartheta}\xspace, the two means of g and h, and the three entries of each of the covariance matrices of (g, h) and of (\eta, \ensuremath{\varepsilon}\xspace)—nine in total.

We add now a different aspect. There are w_1,\dots,w_n, w_i\in\{0,1\}, and the second patient, (\tilde X_i, \tilde Y_i) is observed only if w_i=1. The g_is and h_is are related to the quality of the treatment and the characterization of the patients. The ws on the other hand, are related to the size of the relevant population in the hospital area. Recall that the identity of the hospital is concealed from the data analyzer, and therefore, the S-Bayesian has no much choice than to assume a priori that the ws are independent of the (g,h)s. He must admit, this is a description of his knowledge (AKA, ignorance). In reality, there may be a small spurious correlation between them, which he sincerely believe that is negligible. Note that the distribution of (X,Y) has five observable but still the same nine parameters—a substantial overparametrization.

The conservative frequentist insists on being unbiased, would still have no real difficulty. He proceeds as before using only the observed pairs, i.e., the sub-sample with w_i=1. The Bayesian, on the other hand, have no reason to ignore the sub-sample with w_i=0. Since, by his prior, it is just a random sub-sample and he can use it to learn about the joint distribution of (g,h), which, by his prior, is going to improve the utilization of the sub-samples without missing observations, [IHmodel].

We consider now a toy problem—the white noise model. It can be argued that it is isomorphic to the density estimation problem described above (e.g., using harmonic expansion of the density). However, it is conceptually simple. Suppose X_i\thicksim \ensuremath{{\mathcal{N}}}\xspace(\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i,n^{-1}), i=1,2,\dots,, independent. Practically, as common today, their number is bounded by some p\gg n. That is, at stage n we observe essentially an infinite number of random constants with independent added noise. We assume that, for sure, |\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i|<c_i. We want to estimate \ensuremath{\vartheta}\xspace=\sum\ensuremath{\alpha}\xspace_i\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i, for some constants \ensuremath{\alpha}\xspace_i>0 such that \sum_{i=1}^{\ensuremath{\infty}\xspace}\ensuremath{\alpha}\xspace_i=\ensuremath{\infty}\xspace. However, both \sum_i\ensuremath{\alpha}\xspace_i^2<\ensuremath{\infty}\xspace and \sum_{i=1}^{\ensuremath{\infty}\xspace}|\ensuremath{\alpha}\xspace_ic_i|<\ensuremath{\infty}\xspace. There is nothing that tell us how the \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_is are related to each other, and, admittedly, they are a priori free, independent variables. Moreover, \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_1/c_1,\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_2/c_2,\dots are, to the best of our knowledge, exchangeable and symmetric about 0.

The loss function is the standard quadratic loss. That is, the Bayes estimator is the expected value. Since the expectation of a sum is the sum of the expectations, \ensuremath{\vartheta}\xspace_n^B=\sum_{i=1}^{\ensuremath{\infty}\xspace}\ensuremath{\alpha}\xspace_i\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i^B, where \hat\ensuremath{\vartheta}\xspace^B,\hat\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_1^B,\hat\ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_2^B,\dots are the Bayes estimator of the corresponding parameters, i.e., the a posteriori expectation. Similarly, the typical frequentist is using \sum_{i=1}^{p}\ensuremath{\alpha}\xspace_iX_i, or, maybe, \sum_{i=1}^{p}\ensuremath{\alpha}\xspace_i[X_i]^{c_i}_{c_i}, where [x]_a^b is x truncated to the interval [a,b].

But now, consider a more difficult case: \ensuremath{\alpha}\xspace_i=1/i^\xi and c_i=1/i^\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace, for \xi<1/2 and \ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace+\xi>1. The typical frequentist would use the estimator \hat\ensuremath{\vartheta}\xspace_n=\sum_{i=1}^{m}\ensuremath{\alpha}\xspace_i X_i for m\approx n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1)}. It has a variance n\sum_{i=1}^{m}\ensuremath{\alpha}\xspace_i^2 and its bias-squared is equal to \bigl(\sum_{i=m}^{\ensuremath{\infty}\xspace}|\ensuremath{\alpha}\xspace_ic_i|\bigr)^2, both of them of order n^{-1+\frac{1-2\xi}{2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1}}. It is less than the parametric rate of 1/n, but it can be argue that this is the minimax rate.

The conservative S-Bayesian, who sticks to his beliefs, would use the same prior as before. Note that for i\gg n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace)}, the prior distribution of \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i, which necessarily is supported on [-c_i,c_i] is much more informative about \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i than X_i, whose error is of order n^{-1/2}\gg c_i. Thus, his estimator of \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i hardly depends on the value of X_i. Compare it to the above mentioned typical frequentist who estimates \ensuremath{\vartheta}\xspace using X_i all the way up to n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1)}\gg n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace)}. It may seem that the Bayesian is more sensible.

However, this is not the case. The frequentist is afraid from the bias more than he worries about the variance. Bias accumulates when we add the weighted estimates of the \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_is. Zero mean noise, by nature, accumulate only in the mean square, since there is partial cancelation between periods when the noise is positive and periods when the noice is negative. The Bayesian by adding the terms between n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace)} to n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1)} may add a bias of order n^{-1+\frac{1-2\xi}{2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace}} which is larger than the frequentist typical error. This happens since, for the Bayesian, \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i itself is a random variables, and adding the above mentioned terms for the Bayesian is exactly like adding noise for the frequentist—the terms partially cancel each other.

But, we know that there are other priors, and there are Bayes estimators which are as good as the estimator used by the frequentist. For example, suppose that other the prior, \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i = \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace^*_i \ensuremath{\alpha}\xspace_i, for m^*=n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace)}<i<n^{1/(2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1)}=m and \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace_i^* i.i.d. with some prior distribution on [-n^{-\frac{\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-\xi}{2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1}},+n^{-\frac{\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-\xi}{2\ensuremath{\ensuremath{\ensuremath{\ensuremath{\nu}\xspace}\xspace}\xspace}\xspace-1}}]. If this is the prior, then T=\sum_{i=m^*}^{m}\ensuremath{\alpha}\xspace_i X_i is a sufficient statistics for \ensuremath{\ensuremath{\ensuremath{\ensuremath{\mu}\xspace}\xspace}\xspace}\xspace, and therefore the Bayesian is going to use the same statistics as the frequentist. Note that this prior is different from the prior discussed above, and it depends heavily on the specific functional to be estimated.

I leave it to the reader what world view is represented by this prior.

Consider the same model as in Example 2.2.1. There is a simple Bayesian fix. If under this prior (g_i,h_i,\ensuremath{\varepsilon}\xspace_i,\eta_i) has a distribution that depends on the value of w. As a result, the sub-sample, \{i:\;w_i=0\} has no information relevant to \ensuremath{\vartheta}\xspace. Most F-Bayesians feel good about such a prior. It it true that it does not express their world-view, but it protects well against spurious correlation between w and (g,h). But, this just proves our point.

We try now to examplify that the rational prior probability may not describe a world-view but is defined by the best (frequentist) performance of the resulting decision procedure. We consider a simple model in which there are n unknown parameters p_1,\dots,p_n, (essentially) unrelated and known sampling weights w_1,\dots,w_n. The sampling weights defined independent sample indicators S_1,\dots,S_n, P(S_i=1)=w_i. If S_i=1, we observe a Bernoulli random variable Y_i with P(Y_i=1)=p_i. The target is to estimate the mean of the p_is.

There are two immediate estimators. The simpler one, and the one we claim the S-Bayesian should approximately use, is the mean of the observed Y_is. The other is the modified Horovitz-Thompson estimator, which is semiparametrically efficient. This estimator is a weighted average of the observed Y_is with weights inversely proportionally to the sampling probabilities w_i. The latter estimator is asymptotically unbiased. The estimator used by the S-Bayesian will have a somewhat smaller variance, but it fails miserably being biased if the the prior assumptions do not hold and the weights and the probabilities have some small moderate (unexplained) correlation. It can be argued that in high dimensions, the Horovitz-Thompson estimator is the rational procedure. Now, it is certainly possible to have a formal prior which generates the frequentist estimator. Some were suggested in the literature. But these formal priors would strongly deviate from the model assumptions (which both the Bayesian and the frequentist strongly believe to be, at least approximately, valid). This estimator works, but does not represent subjective beliefs about the value of the unknown parameters. Thus, we argue, in this model, Bayesian computations can be used but the Bayesian formulation lack the methodological foundations.

The actual argument is tedious. See Appendix ? for the necessary details of this example.

The bread and butter of the modern statistical analysis is the ultra-high dimensional linear model: Y_i=\beta^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em} X_i+\ensuremath{\varepsilon}\xspace_i, i=1,\dots,n, where \ensuremath{\varepsilon}\xspace_i is independent of X_i=(X_{i1},\dots,X_{ip})^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em}, p\gg n. The dimension of p can be as high as almost being exponential in n, so also the theoretical asymptotic is, necessarily, with p=p_n, the dependency is vague as long as p_n\gg n, and in the discussion below, we would consider p in some sense as fixed: n as changing, but in the range where it is much smaller than p and p is hardly changing. The goal is estimating \beta, either by itself, with the loss function L(\beta,\hat\beta)=\|\hat\beta-\beta\|^2=\sum_{j=1}^{p}(\hat\beta_j-\beta_j)^2, or as a predictor of Y: L(\beta,\hat\beta)=n^{-1}\sum_{i=1}^{n}(\hat Y_i-Y_i)^2, where \hat Y_i=\hat\beta^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em} X_i. To simplify the discussion further, we may consider a toy version of it that retain the essense of the general model: uhdr* uhdr<???> uhdrnn uhdr uhdr

src:.tex

src:.tex

src:.texalign*

For every n: W_nj &= _j+ n^-1/2 _j, j=1,…,p, _1,_2,… i.i.d. N(0,1).

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

For every n: W_nj &= _j+ n^-1/2 _j, j=1,…,p, _1,_2,… i.i.d. N(0,1).

The target now is estimating \beta_1,\dots,\beta_p under the same quadratic loss function: L(\beta,\hat \beta)=\sum_{j=1}^{p}(\hat\beta_j-\beta_j)^2. We assume that \sum_{j=1}^{p}\beta_j^2<\ensuremath{\infty}\xspace

Since p\gg n, a simple fit of the model to the data yields nonsense. In the regression model the model will fit the noisy data exactly, and the error in the toy version would be p/n\to\ensuremath{\infty}\xspace. The current standard approach has two elements. First, a priori (in the Bayesian or frequentist sense) all the \beta coefficients are equivalent. We do not have a reason to assume that one of them is larger than the other. But, the second point is, we have a reasonably good, low dimension sub-model. I.e., we can tactically assume that most coefficients are zero. The question is which of them, since a priori any small subset could be the best small set of explanatory variables.

There is a standard estimating procedure under stronger than necessary conditions on the design matrix, n^{-1}\sum_{i=1}^{n}X_iX_i^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em}, namely the lasso, suggested first by [Tibshirani1996] and analyzed first in the context of ultra-high dimension analysis in [GreenshteinRitov2004]. In its penalized version: ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

_L &= arg min_ {_i=1^n(Y_i-^ T X_i)^2+_j=1^p|_j|},

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

_L &= arg min_ {_i=1^n(Y_i-^ T X_i)^2+_j=1^p|_j|},

where |\beta|=\sum_{i=1}^{p}|\beta_i|. This may look like a MAP (maximum a posteriori) estimator, as mention by [Tibshirani1996] and argued in [Gribonval]. Formally, indeed, it is the maximum of the posterior if the model is ? with \ensuremath{\varepsilon}\xspace_i\thicksim\ensuremath{{\mathcal{N}}}\xspace(0,\ensuremath{\sigma}\xspace^2), i.i.d., independent of the X_is, and under the prior, the \beta_js are i.i.d. with double-exponential distribution whose mean is \ensuremath{\sigma}\xspace^2/\lambda. There is a debate whether the MAP is a Bayesian procedure, it certainly does not follow the utility maximizer point-of-view for continuous parameter space. A standard Bayesian analysis was suggested by [CastiloEtAl]. Usually it is done using a spike-and-slab prior. Consider the toy version for simplicity. The lasso solution for the toy version is tve* tve<???> tvenn tve tve

src:.tex

src:.tex

src:.texalign*

_L &= arg min_ {_i=1^n(W_j-_j)^2+_j=1^p|_j|}
_Lj &= arg min__j {(W_j-_j)^2+|_j|}
&=

src:.tex

src:.tex

src:.texcases W_j-/2, & W_j>/2
0& |W_j|/2
W_j +/2 & W_j<-/2.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

_L &= arg min_ {_i=1^n(W_j-_j)^2+_j=1^p|_j|}
_Lj &= arg min__j {(W_j-_j)^2+|_j|}
&=

src:.tex

src:.tex

src:.texcases W_j-/2, & W_j>/2
0& |W_j|/2
W_j +/2 & W_j<-/2.

The spike-and-slab prior is a prior that considers the \beta_js as i.i.d., with probability 1-\gamma that \beta_j=0, and, by the prior, if \beta_j\ne 0, then it is taken from some distribution, say, \ensuremath{{\mathcal{N}}}\xspace(0,\tau^2). The a posteriori distribution is that ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

(_j=0W_j) &= (1-)12e^-nW_j^2/2 (1-)12e^-nW_j^2/2+ 12(n^-1+^2) e^-W_j^2/2(n^-1 +^2)
&= 11+ 1- 1n^-1+^2 e^nW_j^22 n^21+n^2 ,
(_jW_j) &= 2^2/n e^-_j^2/2^2 e^ -n(_j-W_j)^2/2 (1-)12e^-nW_j^2/2+ 12(n^-1+^2) e^-W_j^2/2(n^-1 +^2)
&= (_j0W_j) 1+n^2^2 e^-12(n+^-2)(_j - n^2n^2+1W_j )^2.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

(_j=0W_j) &= (1-)12e^-nW_j^2/2 (1-)12e^-nW_j^2/2+ 12(n^-1+^2) e^-W_j^2/2(n^-1 +^2)
&= 11+ 1- 1n^-1+^2 e^nW_j^22 n^21+n^2 ,
(_jW_j) &= 2^2/n e^-_j^2/2^2 e^ -n(_j-W_j)^2/2 (1-)12e^-nW_j^2/2+ 12(n^-1+^2) e^-W_j^2/2(n^-1 +^2)
&= (_j0W_j) 1+n^2^2 e^-12(n+^-2)(_j - n^2n^2+1W_j )^2.

Thus, the a posteriori distribution has a point mass at zero, and otherwise it has \ensuremath{{\mathcal{N}}}\xspace(\frac{n\tau^2}{1+n\tau^2}W_j, \frac{\tau^2}{1+n\tau^2} ). If \gamma and \tau were fixed in n, the sample size is large relative to the a priori slab variance and most of the \beta_js are 0: ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

(_j=0W_j) &11+e^nW_j^2/2
(_jW_j) &11+e^-nW_j^2/2 ne^-n(_j-W_j)^2/2.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

(_j=0W_j) &11+e^nW_j^2/2
(_jW_j) &11+e^-nW_j^2/2 ne^-n(_j-W_j)^2/2.

The Bayesian estimator is a smooth version of the lasso estimator of ? with \ensuremath{\lambda}\xspace\approx \sqrt{2|\log\gamma|/n}.

Now, we don’t say that the Bayesian techniques are useless as a computation tools. We do not argue against a statistician that find a good estimator and then a prior which generates it. We try to argue that doing this is not within the S-Bayesian paradigm that the prior expresses prior knowledge. A particular demand from a priori statement (i.e., a statement that is done before the sample is known) is that parameters of the prior would not depend on the sample size. In particular \gamma and \tau^2 should be fixed. In fact, if we assume that \sum_{j=1}^{p}\beta_j^2\approx c, then \gamma p\tau^2\approx c. In particular, \gamma p, the predicted number of non-zero terms does not grow with n. This is in contrast to the standard assumptions in the sparse model estimation, that the number of non-zero terms is growing with n. For example, if \beta_{(1)},\dots, \beta_{(p)} are \beta_1,\dots,\beta ordered from the largest to the smallest absolute value, then a standard assumption is that \beta_{(j)}=0 for j>n/\log(n), or |\beta_{(j)}|<j^{\ensuremath{\alpha}\xspace} for some \ensuremath{\alpha}\xspace>1/2. In the latter case, we consider the models with M=n^{1/(1+2\ensuremath{\alpha}\xspace)} non-zero terms. The frequentist is not oblige to any prior, his model is tentative and maximize what can be learn from the data. For example, [GreenshteinRitov2004, RigolletTsybakov2012] consider the best model with M=M_n non-zeros coefficients. If we consider \beta_{(j)}\approx j^{-\ensuremath{\alpha}\xspace}, the optimal model size to fit the toy version minimizes the bias contribution of a small model, \approx \sum_{k=M}^{p}k^{-2\ensuremath{\alpha}\xspace}\approx M^{-2\ensuremath{\alpha}\xspace+1} and the estimation error, M/n, thus, M_n\approx n^{1/(2\ensuremath{\alpha}\xspace)}. Thus the optimal value of \ensuremath{\lambda}\xspace in ? satisfies ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

n^1/(2)p &= 2(1- (n/2)) 42ne^-n^2/8, implying &8(( p)-(n)/2)n.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

n^1/(2)p &= 2(1- (n/2)) 42ne^-n^2/8, implying &8(( p)-(n)/2)n.

The dependency of \ensuremath{\lambda}\xspace on n is weak, but is necessary for having a good estimator. In practice, the fine tuning of \ensuremath{\lambda}\xspace can be done using a cross-validation-like procedure.

The frequentist may consider, as above, that j^{\ensuremath{\alpha}\xspace}|\beta_{(j)}| is bounded. The \beta_j are exchangeable, can they be considered as a simple sample? That is, can we find a distribution such that its empirical distribution is \ensuremath{{\mathbb{P}}}\xspace_n(\xi) \approx 1 - c/(p\xi^{1/\ensuremath{\alpha}\xspace})? Not directly. Of course, one can generate \tilde\beta_j \thicksim \ensuremath{{\mathcal{N}}}\xspace(0,j^{-2\ensuremath{\alpha}\xspace}), j=1,2,\dots,p, i.i.d., and then take a random permutation of them. The spike-and-slab prior tries and fails to generate something like that. The non-zero terms are too big and their numbers does not grow with n. The prior of [CastiloEtAl] has two independent components. The number of non-zero terms and their distributions. The prior of the number of components depends on p, which, by itself is problematic. More problematic, is that the prior is constructed such that it would give a reasonably large number of non-zero terms as a posterior, while by the prior itself, their number is actually pretty small, and is bounded with high probability at a bound that depends on p. See Technical Note ?.

src:.tex

src:.tex

src:.tex

src:

src:.tex

src:.tex

src:.tex

src:

Consider estimating \ensuremath{\vartheta}\xspace\triangleq\int f^2(x)dx where f is the common density of n i.i.d. observations. For simplicity, we assume that f\in\ensuremath{{\mathcal{F}}}\xspace_\ensuremath{\alpha}\xspace, where \ensuremath{{\mathcal{F}}}\xspace_\ensuremath{\alpha}\xspace is the set of all densities supported on the unit interval, bounded away from 0 and can be rough but there is c<\ensuremath{\infty}\xspace such that for any x,x+h\in(0,1): |f(x+h)-f(x)|<c|h|^{\ensuremath{\alpha}\xspace}. We consider the case 1/4<\ensuremath{\alpha}\xspace<1/2. It was a surprise that \ensuremath{\vartheta}\xspace can be estimated on the \sqrt n rate. After all, f^2(x) at a given point x can be estimated at a nonparametric rate, when the bias and variance are balanced, and thus, a plug-in estimator of any reasonable estimator is not expected to yield the parametric rate. The semiparametric analysis of this model can be found, for example, in [RitovBickel1990, BickelRitov1988, CaiLow2005, donoho1991geometrizing, birge1995estimation, EfromovichLow1996]. For completeness we give an analysis of a simple estimator.

The efficient procedure can be described schematically as follows. We start with an initial estimator \tilde f(x) of the density. It can be based on a sub-sample. We use the expension sq0* sq0<???> sq0nn sq0 sq0

src:.tex

src:.tex

src:.texalign*

f^2(x)dx &= 2f(x) f(x)dx -f^2(x)dx + (f(x)-f(x))^2 dx

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

f^2(x)dx &= 2f(x) f(x)dx -f^2(x)dx + (f(x)-f(x))^2 dx

The first term is estimated easily by 2n^{-1}\sum_{i=1}^{n}\tilde f(X_i). The second term can be calculated easily, and, hopefully, the third term is small and can be ignored. Here is a detailed simple construction. The asymptotic distribution of the efficient estimator is given in ? below. We start with dividing the sample into two equal random sub-samples and consider the histogram estimators of the density, \tilde f_1(x),\tilde f_2(x) with M=M_n\to\ensuremath{\infty}\xspace bins with equal width, where sqRange* sqRange<???> sqRangenn sqRange sqRange

src:.tex

src:.tex

src:.texalign*

n^1/4M_nn.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

n^1/4M_nn.

Denote the value of the histograms by \tilde f_{jm}, j=1,2, m=1,\dots,M It is then considers ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

&1M_m=1^M (2f_1mf_2m-12f_1m^2-12f_2m^2) + Mn.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

&1M_m=1^M (2f_1mf_2m-12f_1m^2-12f_2m^2) + Mn.

To justify the estimator write \tilde f_{jm}=\bar f_m+\ensuremath{\delta}\xspace_{jm} where \bar f_m is the mean true density in the mth interval. Then sq1* sq1<???> sq1nn sq1 sq1

src:.tex

src:.tex

src:.texalign*

&= 1M _m=1^Mf_j^2 + 1M_m=1^Mf_j(_1m+_2m) + 1M_m=1^M(2_1m_2m- 12_1m^2 -12_2m^2) - M2n

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

&= 1M _m=1^Mf_j^2 + 1M_m=1^Mf_j(_1m+_2m) + 1M_m=1^M(2_1m_2m- 12_1m^2 -12_2m^2) - M2n

The first term in ? is deterministic, let \bar f be the histogram approximation of the density sq1a* sq1a<???> sq1ann sq1a sq1a

src:.tex

src:.tex

src:.texalign*

1M _m=1^Mf_j^2 &= f^2(x) dx
&= f^2(x)dx - 2f(x)( f(x)-f(x))dx -(f(x)-f(x))^2dx
&= f^2(x)dx -(f(x)-f(x))^2dx
&= +O(M^-2)=+ o(n^-1/2).

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

1M _m=1^Mf_j^2 &= f^2(x) dx
&= f^2(x)dx - 2f(x)( f(x)-f(x))dx -(f(x)-f(x))^2dx
&= f^2(x)dx -(f(x)-f(x))^2dx
&= +O(M^-2)=+ o(n^-1/2).

The \ensuremath{\delta}\xspace_{jm} are mean 0, asymptotically uncorrelated, and sq2* sq2<???> sq2nn sq2 sq2

src:.tex

src:.tex

src:.texalign*

n2Mf_j E_jm^2 1.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

n2Mf_j E_jm^2 1.

Thus, the second term of ? satisfies: sq1b* sq1b<???> sq1bnn sq1b sq1b

src:.tex

src:.tex

src:.texalign*

nM_m=1^Mf_j(_1m+_2m)=nM_m=1^M(f_j-)(_1m+_2m)D N(0, 4Var(f(X))).

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

nM_m=1^Mf_j(_1m+_2m)=nM_m=1^M(f_j-)(_1m+_2m)D N(0, 4Var(f(X))).

Finally, we consider the third and fourth terms in ?. We use the fact that M^{-1}\sum_{m=1}^{M}\bar f_m=1, and write: sq1c* sq1c<???> sq1cnn sq1c sq1c

src:.tex

src:.tex

src:.texalign*

1M_m=1^M(2_1m_2m- 12_1m^2 -12_2m^2-f_j M2n) = O_p(M^1/2n) =o_p(n^-1/2) ,

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

1M_m=1^M(2_1m_2m- 12_1m^2 -12_2m^2-f_j M2n) = O_p(M^1/2n) =o_p(n^-1/2) ,

since by ?, this is a sum of asymptotically uncorrelated and mean zero terms, where the terms are \operatorname{O_p}\left(M/n\right). It follows that from ?, ?, ?, and ?: sqeff* sqeff<???> sqeffnn sqeff sqeff

src:.tex

src:.tex

src:.texalign*

n(- )D N(0, 4Var(f(X))),

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

n(- )D N(0, 4Var(f(X))),

which is the semiparametric efficiency bound, [RitovBickel1990].

It is interesting to note that the permitted range of M_n, as given in ?, does not include the bandwidth of an estimator with minimax rate of converges of the density at all, n^{1/(1+2\ensuremath{\alpha}\xspace)}—the frequentist has no difficulties with having a different estimator for estimating the density and for estimating a functional of it.

The (S-)Bayesian has no much of choice. For him, f(x) is a random variable, and under quadratic loss function he should compute ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

Ef^2(x)dx &= (Ef(x))^2 dx + Var(f(x))dx.

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

Ef^2(x)dx &= (Ef(x))^2 dx + Var(f(x))dx.

Thus, she should compute her density estimate {\rm E}\mathop{\!}\nolimits f(x) and inflate it by its variance estimate. Note that if \ensuremath{\alpha}\xspace>1/2, the a posteriori variance of a good prior is \operatorname{o_p}\left(n^{-1/2}\right), and thus the correction is negligible. However, we are interested in the regime were 1/4<\ensuremath{\alpha}\xspace<1/2, and the correction is essential. In fact, the minimax estimator of f has mean square error (MSE) of order n^{-2\ensuremath{\alpha}\xspace/(1+2\ensuremath{\alpha}\xspace)}\gg n^{-1/2}. Thus, \int \bigl({\rm E}\mathop{\!}\nolimits f(x)\bigr)^2dx has estimation error that is of a bigger order than that of the frequentist, so the exact correction is essential.

But the integral is a linear operator and hence the Bayesian has the plug-in problem: he has to use \hat\ensuremath{\vartheta}\xspace = \int \widehat{\{f^2(x)\}}dx. But, any good estimator of f^2(x) is similar to a kernel estimator with bandwidth \approx n^{-1/(1+2\ensuremath{\alpha}\xspace)}, and has too much error. The frequentist overcomes this difficulty by introducing the term M/n which indeed increases the error, but makes the estimator unbiased. This option does not directly exist within the Bayesian paradigm. Whenever there is a statistical error, the Bayes estimator is necessarily biased, even under the assumed distribution. See Technical Note ?.

src:.tex

src:.tex

src:.tex

src:

src:.tex

src:.tex

src:.tex

src:

We have the following result.

src:.tex

src:.tex

src:.tex

src:

The proof can be found in Appendix ?. The set \ensuremath{{\mathcal{S}}}\xspace is built in the following way. The [0,1] interval is divided into K bins, and the the density is of the form f_0(x)+\sum_{k=0}^{K-1}\xi_k g(x-k/K), where \xi_i\in\{-1,1\} and g is a given function supported on [0,1/K]. The difficulty is that any \xi_i is estimated using only an order n/K observations found in the itn bin, and thus the estimation of \xi_i is necessarily biased. The plug-in property enforces plugging in these bias estimates into a convex function, and thus accumulating bias.

Can the Bayesian modified his prior and achieve the semiparametric bound? Of course he can. However, the needed formal prior would reflect his silent acceptance of the semiparametric analysis of the problem and not his a priori uncertainties. The prior would be formal to yield a good estimator, and would depend on the parameter estimated. Thus he would use different priors for estimating \int f^{2\gamma}(x)w(x)dx depending on \gamma and w(\cdot)—priors that depend on the functional and ignore his subjective beliefs on the underlined model S.

src:.tex

src:.tex

src:.tex

src:

We consider the ultra-high dimension version of the partial linear model, which is the estimation of a low dimensional subset of the regression coefficients. This model was introduced and analyzed from a standard frequentist point of view as discussed by different authors, [ZhangZhang, Zuerich, JavanmardMontanari, CaiGuo] with the compatibility or alike assumption, and in [LawRitovInference] without any of these restrictions on the design matrix. The standard model is iwc0* iwc0<???> iwc0nn iwc0 iwc0

src:.tex

src:.tex

src:.texalign*

Y&= X+ ^ T W + ,

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

Y&= X+ ^ T W + ,

where \ensuremath{\varepsilon}\xspace is independent of (X,W), W,\gamma\in\ensuremath{{\mathbb{R}}}\xspace^{p}, \beta,X\in\ensuremath{{\mathbb{R}}}\xspace, and \beta is the parameter of interest. This model is equivalent to the model: iwc* iwc<???> iwcnn iwc iwc

src:.tex

src:.tex

src:.texalign*

X &= ^ T W+,
Y&= ^ T W+,
&= cov(,)Var(),

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

X &= ^ T W+,
Y&= ^ T W+,
&= cov(,)Var(),

where (\zeta,\xi) are independent of W and \psi=\gamma+\beta\phi. We note that the matrix \boldsymbol{W}=(W_1,\dots,W_n)^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em} is typically ill-defined. The model introduces a few challenges to the Bayesian.

[LawRitovInference] considered the following estimator. For a given (large) m: ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

&= _|M|=m (-1X-W_M^2) _|M|=m(-1X-W_M^2) (X-W_M),
&= _|M|=m (-1Y-W_M^2) _|M|=m(-1Y-W_M^2) (Y-W_M),
&=_i=1^n_i_i _i=1^n_i^2,

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

&= _|M|=m (-1X-W_M^2) _|M|=m(-1X-W_M^2) (X-W_M),
&= _|M|=m (-1Y-W_M^2) _|M|=m(-1Y-W_M^2) (Y-W_M),
&=_i=1^n_i_i _i=1^n_i^2,

where bold denotes the vectorized version of the sample, e.g., \boldsymbol {X}=(X_1,\dots,X_n)^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em} and \boldsymbol{W}=(W_1^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em},\dots,W_n^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em})^{\scriptscriptstyle{\textsf T}} \hspace{-0.15em}. The set \ensuremath{{\mathcal{M}}}\xspace\subseteq\{1,\dots,p\}, and \phi_\ensuremath{{\mathcal{M}}}\xspace and \psi_\ensuremath{{\mathcal{M}}}\xspace are the least square estimate corresponding to the models where all coefficients not in \ensuremath{{\mathcal{M}}}\xspace are set to zero.

In words, for each model of size m, we fit a simple linear regressions of Y annd X on W, and weighted them inversely proportional to their residual sum-of-squares. [LawRitovInference] proved that under some regularity conditions the scaled estimation error, \sqrt{n}(\hat\beta-\beta), is asymptotically normal under some regularity conditions. The two main assumptions are, firstly, the linear model ? can be fitted with a sparse model of cardinality no larger than \sqrt{n}/\log(p), and secondly, \ensuremath{\alpha}\xspace>4\max\bigl(\mathop{\rm Var}\nolimits(\zeta),\mathop{\rm Var}\nolimits(\xi)\bigr). The latter condition is not consistent with a simple Bayesian interpretation, as [LawRitovInference] point out.

Sparse models, which are the bread and butter of the current big data statistics, are very problematic. A major difficulty is that the tail of the regression coefficient may be tiny, much below any detection (of order as small 1/\sqrt{p} while p\gg n) and yet explain totally the dependent variable. Since it is below detection, using the results of any statistical procedure as a description of the reality is based totally on an unverifiable belief. The cautious frequentist has no real problem. The claim isn’t that the estimator is approximating the theoretical best predictor, but only that it approximates the best predictor based on a model of a size m. Cf., [GreenshteinRitov2004, RigolletTsybakov2012], while the underlined assumption is only that there is a sparse model that give a better than nothing predictor. When the target is estimating a single coefficient (or, more generally, a few of them) like \beta in ? or ? the issue is more profound. First, the aim is to achieve the \sqrt{n} rate, which put a further limit on the largest model that can be dealt by (to no more than \sqrt{n}/\log(p) or so while the limit is n/\log(p) in the nonparametric problem). Second, the scientific goal of inference, to test whether X has a direct impact on Y controlling for W is not achieved. At most, one can honestly claim that X has direct effect after linearly controlling for any very small subset of the confounders. Third, the scientist may hope to be able to make a statement about causality, but this is not really possible. The S-Bayesian has the further challenges. For example, his prior expresses a priori beliefs and hence should be the same whether the task is estimating \gamma or \beta in the model ?, although, for example, the optimal frequentist sparsity assumptions must be different.

Consider the simple case where the coordiantes of W are statistically independent. The needed sparsity assumption are strong. Let: ** *<???> *nn * *

src:.tex

src:.tex

src:.texalign*

_(m) &= _|M|<mE(Y-_M^ T W), m=1,…,p, _^c(m)=_(p)-_(m)
_(m) &= _|M|<mE(X-_M^ T W), m=1,…,p, _^c(m)=_(p)-_(m)

src:.tex

src:.tex

src:.texequation

src:.tex

src:.tex

src:.texsplit

_(m) &= _|M|<mE(Y-_M^ T W), m=1,…,p, _^c(m)=_(p)-_(m)
_(m) &= _|M|<mE(X-_M^ T W), m=1,…,p, _^c(m)=_(p)-_(m)

When a Bayesian is facing the model ?, he should believe in two things. First that the X and Y can be explained by sparse model, and the sparsity should be strictly smaller than \sqrt{n} (otherwise, the estimation error of the two residuals would prevent estimating \beta on the \sqrt{n} rate). Second, that the residuals from these two regression when consider as Euclidean vectors are each of order smaller than n^{-1/4} (and thus, the correlation between the residuals will be in order smaller than n^{-1/2}). It makes sense that he may believe that X can be explained by W in a way that satisfies these two conditions. It may sense that he believes that the regression of Y on W has these properties. Suppose he has no reason to assume that there is any relation between the way that W explains X and the way it explains Y. This is what he believes. We cannot argue about that. Unfortunately, this is not enough. Suppose X and Y can be explained with models of size m=n^{-1/3}. Even if the component of W are statistically independent, the coefficients of the model can be all of order m^{-1/2}. However, suppose these models do not overlap. The coefficients of \psi corresponding to those in the active variable in the model of size m of \phi are all of order n^{-7/12}, much below the detection level. There contribution to correlation between the estimated residuals is m\times m^{-1/2}\times n^{-7/12}=n^{-5/12}\gg n^{-1/2}.

Can the S-Bayesian “correct” his prior? As in most examples, the answer is yes, but with the price of stop following the S-Bayesian paradigm. If you consider the sampling procedure, evaluate the statistical properties of different estimators, consider which estimator is consistent and robust, and then look for a prior that generates this prior, you may be a good statistician, but you are not a Bayesian.