Private measures, random walks, and synthetic data

First, let us prove the lemma for the indicator of any single point k \in [n], i.e. for x={\textbf 1}_{\{k\}}. Here we have

First, consider j=1, the only index on level \ell=0. The function \psi_1(t)=1/n trivially satisfies \psi_1(k)=1/n and \norm[0]{\psi_j}_2^2 = 1/n, so we have \lambda(x)_1=1.

Next, consider an index j on some level \ell \ge 1. By construction, any function \psi_j on that level can takes on three values: 0 and \pm 2^\ell/n. Moreover, \psi_j is supported on an interval of length n/2^{\ell-1}, see (?). Hence \norm[0]{\psi_j}_2^2 = 2^{\ell+1}/n, so \abs{\lambda(x)_j} \le 2.

Moreover, the functions \psi_j on any given level have disjoint support. So among all such functions on each level, at most one can make the value \psi_j(k) and thus \lambda(x)_j nonzero. As we just showed, for level 0 this value is 1, and for each subsequent level \ell \in \{1,\ldots,L\}, this value is bounded by 2. Summing over all levels, we conclude that \norm{\lambda(x)}_1 \le 2L+1.

To extend this bound to a general function x \in \ell^2[n], decompose it as x = \sum_{k=1}^n x(k) {\textbf 1}_{\{k\}}. Then, by linearity, \lambda(x) = \sum_{k=1}^n x(k) \lambda({\textbf 1}_{\{k\}}), so

The bound \norm[1]{\lambda({\textbf 1}_{\{k\}})}_1 \le 2L+1 from the first part of the argument completes the proof of the lemma.

We can write Z_1+\cdots+Z_n = \langleZ,u\rangle where u = (1,\ldots,1)^\mathsf{T}. Since \norm[0]{n^{-\frac{1}{p}}u}_p=1 and V is 1-Lipschitz in the \ell^{p} norm, the densities of the random vectors Z+n^{-\frac{1}{p}}u and Z differ by a multiplicative factor of at most e pointwise. Therefore,

Rearranging the terms, we deduce that

which completes the proof.

Using (?), (?), and (?), we obtain

The lemma is proved.

Take a measure \mu on [0,1], preprocess it by quantizing as in the previous subsection, and feed the quantized measure \mu_\mathcal{N} into the algorithm \mathcal{B} of Proposition 5.2 for \Omega=\mathcal{N}.

The contraction property (Lemma 5.3) ensures that

This and the privacy property of Proposition 5.2 for measures on \mathcal{N} guarantee that quantization does not destroy privacy, i.e. the algorithm \mu \mapsto \mathcal{B}(\mu_\mathcal{N}) is still \alpha-metrically private as claimed.

As for the accuracy, Proposition 5.2 for the measure \mu_\mathcal{N} gives

Moreover, the accuracy of quantization (?) states that W_1(\mu,\mu_\mathcal{N}) \le 1/n. By triangle inequality, we conclude that

Taking n to be the largest integer less than or equal to \alpha yields the conclusion of the theorem.

For the lower bound, it is enough to find a spanning tree of G of length bounded by \operatorname{TSP}(G). Consider the minimal tour of G of length \operatorname{TSP}(G) as a subgraph of G. Let T be a spanning tree of the tour. Since the tour contains all vertices of G, so does T, and thus T is a spanning tree of G. Since T is obtained by removing some edges of the tour, the length of T is bounded by of the tour, which is \operatorname{TSP}(G). The lower bound is proved.

For the upper bound, note that dropping any edges of G can only increase the value of TSP. Thus TSP of G is bounded by the TSP of its spanning tree T. Moreover, TSP of any tree T equals twice the sum of lengths of the edges of T. This can be seen by considering the depth-first search tour of T, which starts at the root and explores as deep as possible along each branch before backtracking, see Figure ?.

Step 1: constructing a spanning tree. Let us construct a small spanning tree T_0 of T and use Lemma 6.1. Let \varepsilon_j = 2^{-j}, j \in \mathbb{Z}, and let \mathcal{N}_j be \varepsilon_j-nets of T with cardinalities \abs{\mathcal{N}_j} = N(T,\rho,\varepsilon_j). Since T is finite, we must have \abs{\mathcal{N}_j} = 1 for all sufficiently small j. Let j_0 be the largest integer for which \abs{\mathcal{N}_{j_0}} = 1.

At the root of T_0, let us put a single point that forms the net \mathcal{N}_{j_0}. At the next level, put all the points of the net \mathcal{N}_{j_0+1}, and connect them to the root by edges. The weights of these edges, which are defined as the distances of the points to the root, are all bounded by \varepsilon_{j_0}. At the next level, put all points of the net \mathcal{N}_{j_0+2}, and connect each such point to the closest point in the previous level \mathcal{N}_{j_0+1}. (Break any ties arbitrarily.) Since the latter set is a \varepsilon_{j_0+1}-net, the weights of all these edges are bounded by \varepsilon_{j_0+1}. Repeat these steps until the levels do not grow anymore, i.e. until the level contains all the points in T; see Figure ? for illustration.

If all the nets \mathcal{N}_j that make up the levels of the tree T_0 are disjoint, then T_0 is a spanning tree of T. Assume that this is the case for time being.

Step 2: bounding the length of the tree. For each of the levels j = j_0+1, j_0+2,\ldots, the tree T_0 has \abs{\mathcal{N}_j} edges connecting the points of level j to the level j-1, and each such edge has length (weight) bounded by \varepsilon_{j-1}. So \operatorname{MST}(T) is bounded by the sum of the lengths of the edges of T_0, i.e.

Step 3: bounding the sum by the integral. Our choice \varepsilon_j = 2^{-j} yields \varepsilon_{j-1} = 4(\varepsilon_j-\varepsilon_{j+1}). Moreover, our choice of j_0 yields \abs{\mathcal{N}_j} \ge 2 for all j \ge j_0+1, which implies \abs{\mathcal{N}_j} \le 2 \left( \abs{\mathcal{N}_j}-1 \right) for such j. Therefore

An application of Lemma 6.1 completes the proof.

Step 4: splitting. The argument above assumes that all levels \mathcal{N}_j of the tree T_0 are disjoint. This assumption can be enforced by splitting the points of T. If, for example, a point \omega \in \mathcal{N}_j is also used in \mathcal{N}_k for some k < j, add to T another a replica of \omega – a point \omega' that has zero distance to \omega and the same distances to all other points as \omega. Use \omega in \mathcal{N}_j and \omega' in \mathcal{N}_k. Preprocessing the metric space (T,\rho) by such splitting yields a pseudometric space (T',\rho) in which all levels \mathcal{N}_j are disjoint, and whose TSP is the same.

Applying Folding Proposition 6.5, we obtain an n-element subset \Omega \subset [0,\operatorname{TSP}(T)] and a 1-Lipschitz bijection F: \Omega \to T. Applying Proposition 5.2 and rescaling by the factor \operatorname{TSP}(T), we obtain an \alpha-metrically private algorithm \mathcal{B} that transforms a probability measure \mu on \Omega into a probability measure \nu on \Omega, and whose accuracy is

Define a new metric \bar{\rho} on \Omega by \bar{\rho}(x,y) = \rho \left( F(x),F(y) \right). Since F is 1-Lipschitz, we have \bar{\rho}(x,y) \le \abs{x-y}. Note that the Wasserstein distance can only become smaller if the underlying metric is replaced by a smaller metric. Therefore, the bound (Proof), which holds with respect to the usual metric \abs{x-y} on \Omega, automatically holds with respect to the smaller metric \bar{\rho}(x,y).

It remains to note that (\Omega, \bar{\rho}) is isometric to (T,\rho). So the accuracy result (Proof), which as we saw holds in (\Omega, \bar{\rho}), automatically transfers to (T,\rho) (by considering the pushforward measure).

Preprocess the input measure \mu by quantizing as in the previous subsection, and feed the quantized measure \mu_\mathcal{N} into the algorithm \mathcal{B} of Proposition 7.1 for the metric space (\mathcal{N},\rho).

The contraction property (?) ensures that

for any two input measures \mu and \mu'. This and the privacy property in Proposition 7.1 for measures on \mathcal{N} guarantee that quantization does not destroy privacy, i.e. the algorithm \mathcal{A}: \mu \mapsto \mathcal{B}(\mu_\mathcal{N}) is still \alpha-metrically private as claimed.

Next, the accuracy property in Proposition 7.1 for the measure \mu_\mathcal{N} on \mathcal{N} gives

Moreover, the accuracy of quantization (?) states that W_1(\mu,\mu_\mathcal{N}) \le \delta. By triangle inequality, we conclude that

Thus, by (?),

Since N(T,\rho,2\delta)\leq N(T,\rho,\delta), replacing \delta by 2\delta completes the proof of the theorem.

Since the output probability measure \nu in Theorem 7.2 is finitely supported, it has the form

for some natural number r, positive weights w_i and elements R_i \in T.

Let us quantize the weights w_i by the uniform quantizer with step 1/m where m is a large integer. Namely, set

Obviously, the total quantization error satisfies

To make the quantized weights a probability measure, let us add the total quantization error to any given weight, say the first. Thus, define

and set

Note the three key properties of \nu'. First, since the weights w'_i sum to one, \nu' is a probability measure. Second, since \nu' is obtained from \nu by transporting a total mass of \kappa across the metric space T, we have

where the second inequality follows from (?) and the last one by choosing m large enough. Third, all quantized weights q(w_i) belong to \frac{1}{m} \mathbb{Z} by definition. Thus, \kappa = 1 - \sum_{i=1}^r q(w_i) is also in \frac{1}{m} \mathbb{Z}. Therefore, all weights w'_i are in \frac{1}{m} \mathbb{Z}, too. Hence, w'_i = m_i/m for some nonnegative integers m_i. In other words,

Since \nu' is a probability measure, we must have \sum_{i=1}^r m_i = m. Redefine the sequence Y_1,\ldots,Y_m by repeating each element Y_i of the sequence Y_1,\ldots,Y_r exactly m_i times. Thus \nu' = \frac{1}{m} \sum_{i=1}^m \delta_{Y_i}, as required.

For contradiction, assume that

for all x\in\mathcal{M}_0. Let \mathcal{N} be a t-separated subset of the metric space (\mathcal{M}_0, \rho_1) with cardinality

The separation condition implies that the balls B(y,\rho_1, t/2) centered at the points y \in \mathcal{N} and with radii t/2 are all disjoint.

Fix any reference point y \in \mathcal{M}_0. The disjointness of the balls yields

On the other hand, by the definition of \alpha-metric privacy, for each x \in \mathcal{N} we have:

The diameter assumption yields \rho_0(x,y) \le 1. Furthermore, using the assumption (Proof) and Markov’s inequality, we obtain

Combining the two bounds gives

Substitute this into (?) to get

In other words, we conclude that \abs{\mathcal{N}} \le 2e^\alpha, which contradicts (Proof). The proof is complete.

Let \mathcal{N} \subset T be a t-separated subset of cardinality \abs{\mathcal{N}} = {N_\textrm{pack}}(T,\rho,t). Lemma 8.4 implies the existence of a set of at least \exp(c \abs{\mathcal{N}}) probability measures on T that is (t/3)-separated in the Wasserstein distance. In other words, we have {N_\textrm{pack}} \left( \mathcal{M}(T), W_1, t/3 \right) \ge \exp(c \abs{\mathcal{N}}). Proposition 8.2 is proved.

Suppose that \gamma is a coupling of \mu and \nu. Since \nu is supported on B, we have \gamma(B^c \times B^c) \le \gamma(T \times B^c) = \nu(B^c) = 0, which means that \gamma(B^c \times B^c)=0. Therefore

Since the sets B^c and B are disjoint, the separation assumption implies that \rho(x,y) > t for all pairs x\in B^{c} and y\in B. Thus,

Since this holds for all coupling \gamma of \mu and \nu, the result follows.

Let \mu = n^{-1} \sum_{i=1}^n \delta_{X_i} and \nu = n^{-1} \sum_{i=1}^n \delta_{Y_i} be two independent random empirical measures on T. Let us condition on \nu and denote B = \operatorname{supp}(\nu). Then

Now, {\textbf 1}_{\{X_i \in B^c\}} are i.i.d. Bernoulli random variables that take value 1 with probability

since by construction we have \abs{B} \le n and by assumption \abs{\mathcal{N}} \ge 2n. Then, applying Chernoff inequality (see [vershyninbook, Exercise 2.3.2]), we conclude that \mu(B^c) > 1/3 with probability bigger than 1-e^{-5cn}, where c>0 is a universal constant. Lemma 8.3 yields that W_1(\mu,\nu) > t/3.

Now consider a sequence \mu_1,\ldots,\mu_K of independent random empirical measures on T. Using the result above and taking a union bound we conclude that, with probability at least 1-\binom{K}{2}e^{-5cn}, the inequality W_1(\mu_i,\mu_j) > t/3 holds for all pairs of distinct indices i,j \in \{1,\ldots,K\}. Choosing K = \lceil e^{cn} \rceil makes K between e^{cn} (as claimed) and e^{2cn}. Thus, the success probability is more than 1- (e^{2cn})^2 e^{-5cn}, which is positive. The existence of the required family of measures follows.

The assumption on the packing number for a sufficiently large constant C and Proposition 8.2 yield

Next, apply Proposition 8.1 with t/3 instead of t, and for \mathcal{M}_0 = \mathcal{M}_1 = \mathcal{M}(T), setting \rho_1 and \rho_0 to be the Wasserstein and the TV metrics, respectively. The required conclusion follows.

First note that a version of Proposition 8.2 holds for empirical measures. Namely, denote the set of all empirical measures on n points of T by \mathcal{M}_n(T). If {N_\textrm{pack}}(T,\rho,t) > 2n then we claim that

To see this, let \mathcal{N} \subset T be a t-separated subset of cardinality \abs{\mathcal{N}} > 2n. Lemma 8.4 implies the existence of a set of at least e^{cn} \ge 2e^{c_1n} members of \mathcal{M}_n(T) that is (t/3)-separated in the Wasserstein distance. The claim (Proof) follows.

In preparation to apply Proposition 8.1, consider the sets \mathcal{M}_0 \coloneqq T^n and \mathcal{M}_1 \coloneqq \cup_{k=1}^\infty T^k. Consider the normalized Hamming metric

on \mathcal{M}_0, and the Wasserstein metric

on \mathcal{M}_1. Then we clearly have \operatorname{diam}(\mathcal{M}_0,\rho_0) \le 1, and (Proof) is equivalent to {N_\textrm{pack}}(\mathcal{M}_0,\rho_1,t/3) > 2e^{c_1 n}.

If \mathcal{A}: \mathcal{M}_0 \to \mathcal{M}_1 is a c-differentially private algorithm, then \mathcal{A} is (cn)-metrically private in the metric \rho_0 due to Lemma 2.3. Applying Proposition 8.1 with t/3 instead of t and \alpha = c_1n, we obtain the required conclusion.

We deduce the upper bound from Theorem 7.2 and the lower bound from Theorem 8.5.

Upper bound. By rescaling, we can assume without loss of generality that \operatorname{diam}(T,\rho) = 1. Fix any \varepsilon >0. By definition of Minkowski dimension, there exists \delta_0>0 such that

Then

where K = 1/(d+\varepsilon-1) and I(\delta_0) = \int_{\delta_0}^1 N(T,\rho,x) \, dx. The last step follows if we replace \delta_0 by infinity and compute the integral.

If we let \delta \downarrow 0, we see that K(1/\delta)^{d+\varepsilon-1} \to \infty while I(\delta_0) stays the same since it does not depend on \delta. Therefore, there exists \delta_1>0 such that I(\delta_0) \le K(1/\delta)^{d+\varepsilon-1} for all \delta \in (0,\delta_1). Therefore,

Applying Theorem 7.2 for such \delta and using (Proof), we get

Optimizing in \delta, we find that a good choice is

For any sufficiently large \alpha, we have \delta < \min(\delta_0,\delta_1) as required, and substituting \delta = \delta(\alpha) into the bound in (?) we get after simplification:

Furthermore, recalling that K does not depend on \alpha, it is clear that

Thus

Since \varepsilon>0 is arbitrary, it follows that

Lower bound. Fix any \varepsilon>0. By definition of Minkowski dimension and the equivalence (?), there exists \delta_0>0 such that

Set

Then, for any sufficiently large \alpha, we have x \in (0,\delta_0) and

Applying Theorem 8.5, we get

It is easy to check that

Thus

Since \varepsilon>0 is arbitrary, it follows that

Combining with the upper bound (?), we complete the proof.