Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

([BGM23]) If f is L-Lipschitz then \widehat{\operatorname{Gap}}_{\mathsf{SP}} is \sqrt{2}L-Lipschitz.

Let \psi:\mathcal{W}\times\Theta\mapsto\mathbb{R} be \mu-strongly-convex/strongly-concave w.r.t. \|\cdot\|_w and \|\cdot\|_\theta. Then the algorithm which returns the saddle point of (w,\theta) \mapsto \frac{1}{n}\sum_{z\in S} f(w,\theta;z) + \psi(w,\theta) is (\frac{2L}{\mu n})-UAS.

Let \mathcal{A}_{\mathsf{sub}} satisfy \hat{\alpha}-relative accuracy for any (5L)-Lipschitz loss function and dataset of size n'=\frac{n}{\log(n)} and assume \|\cdot\|^2_w and \|\cdot\|^2_\theta are \frac{1}{\kappa}-strongly convex under \|\cdot\|_w and \|\cdot\|_\theta respectively. Then Algorithm ?, run with \mathcal{A}_{\mathsf{sub}} as a subroutine and \lambda=\frac{48}{B}\left({\hat{\alpha}\kappa^2 + \frac{L\kappa^{3/2}}{\sqrt{n'}}}\right), satisfies

Let f:\ensuremath{\mathcal{Z}}\times\ensuremath{\mathcal{X}}\mapsto \mathbb{R} be L-Lipschitz. Let [w^*,\theta^*]\in\ensuremath{\mathcal{Z}} be the population saddle point. For any x\in\ensuremath{\mathcal{X}} define h([w,\theta];x)=f(w,\theta^*;x) - f(w^*,\theta;x). For S\sim \mathcal{D}^n, let H_S(z) = \frac{1}{n}\sum_{x\in S} h(z;x) and H_\mathcal{D}(z) = \underset{x\sim\mathcal{D}}{\mathbb{E}}\left[ h(z;x) \right]. Then for any \Delta-UAS algorithm, \mathcal{A}, one has \underset{S,A}{\mathbb{E}}\left[ H_\mathcal{D}(\mathcal{A}(S))-H_S(\mathcal{A}(S)) \right] \leq 2\Delta L.

Under the setup described above, there exists an (\epsilon,\delta)-DP algorithm which satisfies \hat{\alpha}-relative accuracy with parameter \hat{\alpha}=O\Big(\sqrt{\kappa(B_w^2L_w^2+B_\theta^2L_\theta^2)}\big(\frac{\sqrt{d\log(1/\delta)\tilde{\kappa}}}{n\epsilon} + \frac{1}{\sqrt{n}}\big)\Big) and runs in at most O\big(\min\big\{\frac{\sqrt{\kappa}n^2\epsilon^{1.5}}{\log^2(n)\sqrt{d\log(1/\delta)\tilde{\kappa}}}, \frac{\sqrt{\kappa}n^{3/2}}{\log^{3/2}(n)}\big\}\big) number of gradient evaluations, where \tilde{\kappa}=1+\mathbf{1}\left\{{p<2 \lor q < 2}\right\}\cdot\log(d).

There exists an Algorithm, \mathcal{R}, which is (\epsilon,\delta)-DP, has number of gradient evaluations bounded by O\big(\min\big\{\frac{\sqrt{\kappa}n^2\epsilon^{1.5}}{\log(n)\sqrt{d\log(1/\delta)\tilde{\kappa}}}, \frac{\sqrt{\kappa}n^{3/2}}{\sqrt{\log(n)}}\big\}\big), and satisfies (up to \log(n) factors),

(\kappa is at most \log(d).)

Let \mathcal{A}_{\mathsf{sub}} satisfy \hat{\alpha}-relative stationarity for any (5L)-bounded monotone operator and dataset of size n'=\frac{n}{\log(n)}. Let \rho satisfy Assumption 1. Then Algorithm ?, run with \mathcal{A}_{\mathsf{sub}} as a subroutine and \lambda=\frac{48}{B}\left({\hat{\alpha}\kappa^3 + \frac{(\betaB+L)\kappa^2}{\sqrt{n'}}}\right), satisfies

Let g(z;x) = g_1(z;x) + g_2(z) such that g_1:\ensuremath{\mathcal{Z}}\times\ensuremath{\mathcal{X}}\mapsto\mathbb{R}^d is L-bounded and \beta-Lipschitz with respect to z and g_2:\ensuremath{\mathcal{Z}}\mapsto\mathbb{R}^d is any (data indepedent) operator. Let z^*\in\ensuremath{\mathcal{Z}} be its population equilibrium point. For any x\in\ensuremath{\mathcal{X}}, define h(z;x)=\left\langle g(z;x),z-z^* \right\rangle. For S\sim \mathcal{D}^n, let H_S(z) = \frac{1}{n}\sum_{x\in S} h(z;x) and H_\mathcal{D}(z) = \underset{x\sim\mathcal{D}}{\mathbb{E}}\left[ h(z;x) \right]. Then for any \Delta-UAS algorithm, \mathcal{A}, one has \underset{S,A}{\mathbb{E}}\left[ H_\mathcal{D}(\mathcal{A}(S))-H_S(\mathcal{A}(S)) \right] \leq \Delta(\betaB+L).

There exists an Algorithm, \mathcal{R}, which is (\epsilon,\delta)-DP, has gradient evaluations bounded by O\big(\min\big\{\frac{\sqrt{\kappa}n^2\epsilon^{1.5}}{\log(n)\sqrt{d\log(1/\delta)\tilde{\kappa}}}, \frac{\sqrt{\kappa}n^{3/2}}{\sqrt{\log(n)}}\big\}\big), and satisfies (up to \log(n) factors)

where \kappa = \frac{1}{\max\left\{{p,1+\frac{1}{\log(d)}}\right\}-1} is at most \log(d) and \tilde{\kappa} = 1+\mathbf{1}\left\{{p<2}\right\}\cdot\log(d).

For any z, it holds that \|\nabla (\frac{1}{2}\|z\|^2)\|_* \leq \|z\|.

Let \mu,\lambda > 0 and \rho be \mu-strongly monotone w.r.t. \|\cdot\|. Then for any point z_0\in\ensuremath{\mathcal{Z}}, the algorithm which outputs the unique equilibrium of G_S(z) + \frac{\lambda}{\mu} (\rho(z)-\rho(z_0)) is \big(\frac{2L}{\mu \lambda n}\big)-uniform argument stable w.r.t. S.

There exists a differentiable convex-concave loss, distribution \mathcal{D}, and point z' such that when the operator is the saddle operator of the loss, \operatorname{VI-Gap}(z) < \operatorname{Gap_{\mathsf{SP}}}(z).

(Restatement of Lemma 2) Let f:\ensuremath{\mathcal{Z}}\times\ensuremath{\mathcal{X}}\mapsto \mathbb{R} be L-Lipschitz. Let [w^*,\theta^*]\in\ensuremath{\mathcal{Z}} be the population saddle point. For any x\in\ensuremath{\mathcal{X}} define h([w,\theta];x)=f(w,\theta^*;x) - f(w^*,\theta;x). For S\sim \mathcal{D}^n, let H_S(z) = \frac{1}{n}\sum_{x\in S} h(z;x) and H_\mathcal{D}(z) = \underset{x\sim\mathcal{D}}{\mathbb{E}}\left[ h(z;x) \right]. Then for any \Delta-UAS algorithm, \mathcal{A}, one has \underset{S,A}{\mathbb{E}}\left[ H_\mathcal{D}(\mathcal{A}(S))-H_S(\mathcal{A}(S)) \right] \leq 2\Delta L.

Let \lambda \geq \frac{48L\kappa^{3/2}}{B\sqrt{n'}} and \mathcal{A}_{\mathsf{sub}} be such that for all t\in [T] it holds that \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z^*_{S,t}\right\|^2 \right] \leq \frac{B^2}{12\cdot2^{2t}\kappa^{3/2}}. Then Recursive Regularization satisfies

Let F:\ensuremath{\mathcal{Z}}\mapsto\mathbb{R}^d be a \gamma-SC/SC function and let [w^*,\theta^*] be the saddle point. Then \|[w,\theta]-[w^*,\theta^*]\|^2 \leq \frac{2\left({F(w,\theta^*)-F(\theta^*,w)}\right)}{\lambda}.

Assume the conditions of Theorem 3 hold. Then for every t\in[T], the following holds

1. \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z_{t}^*\right\| \right]^2 \leq \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z_{t}^*\right\|^2 \right] \leq \frac{B^2}{2^{2t}\kappa}; and,

2. B_t^2 := \underset{}{\mathbb{E}}\left[ \left\|z_{t}^* - z^*_{t-1}\right\| \right]^2 \leq \underset{}{\mathbb{E}}\left[ \left\|z_{t}^* - z^*_{t-1}]\right\|^2 \right] \leq \frac{B^2}{2^{2(t-1)}}.

Let \psi:\ensuremath{\mathcal{Z}}\mapsto\mathbb{R} be any non-negative function which is 1-strongly convex w.r.t. \|\cdot\|. Assume \forall t\in[T] that \underset{}{\mathbb{E}}\left[ \ensuremath{\mathcal{O}}(z_t) \right]=G(z_t) and \underset{}{\mathbb{E}}\left[ \|\ensuremath{\mathcal{O}}(z_t) - G(z_t)\|_*^2 \right] \leq \tau^2. Then for for any z\in\ensuremath{\mathcal{Z}} Algorithm ? satisfies

Let \epsilon,\delta \in (0,1] and c be a universal constant. Let D\in\ensuremath{\mathcal{Y}}^n be a dataset over some domain \ensuremath{\mathcal{Y}}, and let h_1,...,h_T:\ensuremath{\mathcal{Y}}\mapsto\mathbb{R}^d be a series of (possibly adaptive) queries such that for any y\in\ensuremath{\mathcal{Y}}, t\in[T], \left\|h_t(y)\right\|_2 \leq L. Let \sigma \geq \frac{c L \sqrt{T\log(1/\delta)}}{n\epsilon} and T \geq \frac{n^2\epsilon}{b^2}. Then the algorithm which samples batches of size B_1,..,B_t of size b uniformly at random and outputs \frac{1}{b}\sum_{y\in B_t}h_t(y) + g_t for all t\in[T] where g_t \sim \mathcal{N}(0,\mathbb{I}_d\sigma^2), is (\epsilon,\delta)-DP.

Let [w_0,\theta_0],[w,\theta] satisfy \underset{}{\mathbb{E}}\left[ \|[w_0,\theta_0]-[w,\theta]\| \right] \leq \hat{D}. Let g:\mathcal{W}\times\Theta \times \ensuremath{\mathcal{X}} \mapsto \mathcal{B}^{d_w}_{\|\cdot\|_{\bar{p}}}(B_w L_w)\times\mathcal{B}^{d_\theta}_{\|\cdot\|_{\bar{q}}}(B_\theta L_\theta) be a monotone operator and G_S(w,\theta)=\frac{1}{n}\sum_{x\in S} g([w,\theta];x). There exists an implementation of \ensuremath{\mathcal{O}} such that Algorithm ? is (\epsilon,\delta)-DP and the following holds

Further, the resulting algorithm makes O\big(\min\big\{\frac{\sqrt{\kappa}n^2\epsilon^{1.5}}{\log^2(n)\sqrt{d\log(1/\delta)\tilde{\kappa}}}, \frac{\sqrt{\kappa}n^{3/2}}{\log^{3/2}(n)}\big\}\big) gradient evaluations.

If g is L-bounded then \widehat{\operatorname{VI-Gap}} is L-Lipschitz.

Let \lambda \geq \frac{48\tilde{L}\kappa^2}{B\sqrt{n'}} and \mathcal{A}_{\mathsf{sub}} be such that for all t\in [T] it holds that \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z^*_{S,t}\right\|^2 \right] \leq \frac{B^2}{12\cdot2^{2t}\kappa^2}. Then Recursive Regularization satisfies

Let G:\ensuremath{\mathcal{Z}}\mapsto\mathbb{R}^d be a \mu-strongly monotone operator and let z^* be the equilibrium point. Then \|z-z^*\|^2 \leq \frac{2\left\langle G(z),z-z^* \right\rangle}{\mu}.

Assume the conditions of Theorem 5 hold. Then for every t\in[T], the following holds

1. \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z_{t}^*\right\| \right]^2 \leq \underset{}{\mathbb{E}}\left[ \left\|\bar{z}_t - z_{t}^*\right\|^2 \right] \leq \frac{B^2}{2^{2t}\kappa^2}; and,

2. B_t^2 := \underset{}{\mathbb{E}}\left[ \left\|z_{t}^* - z^*_{t-1}\right\| \right]^2 \leq \underset{}{\mathbb{E}}\left[ \left\|z_{t}^* - z^*_{t-1}]\right\|^2 \right] \leq \frac{B^2}{2^{2(t-1)}}.

(Implicit in [PB16, Theorem 3]) Let \beta,\mu,K>0 and c a universal constant. Let g:\ensuremath{\mathcal{Z}}\times\ensuremath{\mathcal{X}}\mapsto\mathbb{R} be monotone and \beta-Lipschitz and \rho:\ensuremath{\mathcal{Z}}\mapsto\mathbb{R} a \mu-strongly monotone operator. Let z_S^* be the equilibrium of G_S(z) = \frac{1}{n}\sum_{x\in S} g(z;x) + \rho(z). There exists an algorithm, which in O(n+\sqrt{n}K\frac{\beta}{\mu}) gradient evaluations find a point \bar{z} such that \underset{}{\mathbb{E}}\left[ \|z^*-\bar{z}\|_2 \right] = c B e^{-K}.

Let \mathcal{A}_{\mathsf{sub}} be as described above. Then Algorithm ? is (\epsilon,\delta)-DP and when run with \lambda = \frac{48}{B}\left({\frac{L}{\sqrt{n'}} + \frac{L\sqrt{d\log(2/\delta)}}{n'\epsilon}}\right) satisfies

and runs in at most O(n + \beta n\log(n/\delta)) gradient evaluations.

Let f(z;x)=\left\langle z,x \right\rangle and define the operator g(z;x)=\nabla f(z;x) = x. Then \operatorname{VI-Gap}(z) = F_{\mathcal{D}}(z)-\min_{u\in\ensuremath{\mathcal{Z}}}\left\{{F_{\mathcal{D}}(u)}\right\}. That is, the strong VI-gap w.r.t. g is equal to the excess population risk w.r.t. the f.

([bassily2021non, Theorem 7.1]) Let p\in(1,2) and p^*=\frac{p}{p-1} and \ensuremath{\mathcal{Z}}=\mathcal{B}_{\|\cdot\|_p}(1). Let \epsilon>0 and 0 < \delta < \frac{1}{n^{1+\Omega(1)}} and let f(z;x)=\left\langle z,x \right\rangle. For any (\epsilon,\delta)-DP algorithm \mathcal{A}, there exists a distribution \mathcal{D} over \ensuremath{\mathcal{X}}=\mathcal{B}_{\|\cdot\|_{p^*}}(1) such that