The Information Bottleneck is NP-hard

The identity (2) follows from the law of total probability.

For the converse, we have that P_{T\mid X}(t\mid x)\geq 0 for all x and t, since the right-hand side of (2) is non-negative. Moreover, by (2), we have that

for every x \in \mathcal{X}. Hence, P_{T\mid X} is a valid channel from \mathcal{X} to \mathcal{T}, and since T depends only on X, T is conditionally independent of Y given X. Consequently, P_{T\mid X} is an encoder of X. Moreover, one can also see that for any t\in \mathcal{T}

and that, for t\in \operatorname{supp}(P) and x\in \mathcal{X}, by the Bayes’ rule, we have that

We start by showing that IB-Lagrangian is NP-hard. To this end, we show that the construction of the instance (P_{XY},\tau) of IB-Lagrangian described above is a polynomial-time reduction from Clique to IB-Lagrangian.

Let (G,k) be an instance of Clique satisfying (?). Following the construction in Section ?, the reduction first constructs the extended graph {\bar G} of G, and then computes the joint distribution P_{XY} defined in (?) and a rational threshold \tau satisfying (?). Finally, it outputs the instance (P_{XY}, \tau) of IB-Lagrangian.

We first show that (P_{XY}, \tau) can be constructed in time polynomial in the size of (G,k). Recall that m = k-1, n = |V(G)| and N = |E(G)|, and assume that the graph G is represented by its adjacency matrix. Then the size of (G,k) is at least n, and, therefore, for our aim it suffices to show that the instance of IB-Lagrangian can be constructed in time polynomial in n. We notice that N \le \binom{n}{2} \le n^2, and, by (?), \binom{k}{2} = B \le N \le \binom{n}{2}, which implies k \le n and hence k-1=m < n. The graph {\bar G} is obtained from G by adding m-2 new vertices for each edge of G, together with the edges of the cliques C_e. Therefore {\bar G} has

vertices, and, by Lemma 7,

oriented edges. Therefore, {\bar G} can be constructed in polynomial time in n. The joint distribution P_{XY} is a table with |A({\bar G})| \cdot |V({\bar G})| \le n^4(n + n^3) entries, and, by (?), each entry is either 0 or 1/(2M). Since M \le n^4, each entry is a rational number whose numerator and denominator have O(\log n) bits. Therefore, P_{XY} has polynomial size and can be constructed in polynomial time.

It remains to compute \tau. Note that this does not require computing L_{1/2}(P_{XY}), but only approximating \Lambda_0 + \delta/2, where \Lambda_0 = g_{{\bar G}}(P_X) - c_m is the value of the explicit deterministic encoder T_0 defined in (?). The value \delta = 1/(32 N m^4), chosen in (12), is a rational number, with O(\log n) bits. On the other hand, the value \Lambda_0 is in general irrational, but it is an explicit expression involving polynomially many logarithms of rational numbers. Indeed, we have that H(P_X) = \log M and c_m = \frac12 \log\frac{m}{m-1}. Moreover, let d_v denote the degree of a vertex v in {\bar G}. Then v is an endpoint of 2d_v oriented edges in A({\bar G}), each of which has probability 1/M under P_X and is mapped to v with probability 1/2 by W_{{\bar G}} defined in (?). Hence, we have that (W_{{\bar G}} P_X)(v) = 2d_v \cdot \frac{1}{M} \cdot \frac12 = \frac{d_v}{M}, and

Therefore, by Lemma 15 in Appendix ?, an expression of this form can be approximated by a rational number within any additive error \eta > 0 in time polynomial in the bit length of its coefficients and of 1/\eta. By taking \eta = \delta/12, which has O(\log n) bits, a rational \tau satisfying (?) can be computed in polynomial time. Hence, the whole reduction runs in polynomial time.

We now show that the reduction is correct, that is, the instance (P_{XY}, \tau) of IB-Lagrangian is a yes-instance if and only if the corresponding instance (G,k) of Clique is a yes-instance. Recall that (P_{XY}, \tau) is a yes-instance of IB-Lagrangian if there exists an encoder T of X such that I(Y;T) - \frac12 I(X;T) \ge \tau, and that (G,k) is a yes-instance of Clique if \omega(G) \ge k, that is, G contains a clique of size k.

Note that, by (?), we have that

that is, \tau lies strictly between \Lambda_0 and \Lambda_0 + \delta.

Assume first that (G,k) is a yes-instance, that is, \omega(G) \ge k. By Lemma 12 and the rightmost inequality in (?), the encoder T_1 satisfies

Hence (P_{XY}, \tau) is a yes-instance of IB-Lagrangian. Moreover, L_{1/2}(P_{XY}) > \tau + \frac{5\delta}{12}.

Assume now that (G,k) is a no-instance, that is, \omega(G) < k. Then, by Lemma 9 and the leftmost inequality in (?), every encoder T of X satisfies

Hence, no encoder reaches the threshold \tau, and (P_{XY}, \tau) is a no-instance of IB-Lagrangian. Moreover, L_{1/2}(P_{XY}) \le \tau - \frac{5\delta}{12}.

Therefore, we have shown that the reduction is correct, and, since Clique is NP-hard, also when restricted to instances satisfying (?), IB-Lagrangian is NP-hard.

We now prove the inapproximability result. The previous argument shows that L_{1/2}(P_{XY}) > \tau + \frac{5\delta}{12} if (G,k) is a yes-instance of Clique, and L_{1/2}(P_{XY}) \le \tau - \frac{5\delta}{12} if (G,k) is a no-instance. Hence, any approximation of L_{1/2}(P_{XY}) within an additive error smaller than \frac{5\delta}{12} allows to decide Clique in polynomial time. Suppose, for the sake of contradiction, that there exists a polynomial-time algorithm that, given P_{XY}, outputs a value v such that

By Lemma 13, we have that \frac{1}{216 M^{3/2}} \le \frac{\delta}{3} < \frac{5\delta}{12}. Hence, if \omega(G) \ge k, then v \ge L_{1/2}(P_{XY}) - \frac{\delta}{3} > \tau, while, if \omega(G) < k, then v \le L_{1/2}(P_{XY}) + \frac{\delta}{3} < \tau. Thus, running the reduction, then the algorithm, and comparing v with \tau would decide Clique in polynomial time, which is impossible unless \mathrm{P} = \mathrm{NP}.

We start by showing that IB-Threshold is NP-hard. To this end, we show that the construction of the instance (P_{XY}, R, \vartheta) described above is a polynomial-time reduction from Clique to IB-Threshold.

Let (G,k) be an instance of Clique satisfying (?). The reduction first constructs the extended graph {\bar G} and the joint distribution P_{XY} defined in (?), as in the proof of Theorem 1. Then, it computes a rational number \tilde R with |\tilde R - R^*| \le \frac{\delta}{4}, the budget R in (?), and a rational threshold \vartheta satisfying (?). Finally, it outputs the instance (P_{XY}, R, \vartheta) of IB-Threshold.

We first show that (P_{XY}, R, \vartheta) can be constructed in time polynomial in the size of (G,k). As in the proof of Theorem 1, it suffices to show that the construction runs in time polynomial in n=|V(G)|. We have already shown that {\bar G} and P_{XY} can be constructed in polynomial time in the proof of Theorem 1. Thus, it remains to show that R and \vartheta can be computed in polynomial time. By (14), R^* is a linear combination of the five logarithms \log M, \log(k(k-1)), \log(m(m-1)), \log(1/a_0) and \log(1/a_1), with rational coefficients. Since M \le n^4 and k, m \le n, all these coefficients and the arguments of the logarithms are rational numbers with O(\log n) bits. By Lemma 15 in Appendix ?, R^* can be approximated within additive error \frac{\delta}{4} in polynomial time. Consequently, \tilde R and R can be computed in polynomial time. Finally, \frac{R}{2} + \Lambda_0 + \frac{3\delta}{8} is the sum of the rational number \frac{R}{2} + \frac{3\delta}{8} and of \Lambda_0, which, as shown in the proof of Theorem 1, is an explicit expression involving polynomially many logarithms of rational numbers with O(\log n) bits. Hence, again by Lemma 15 in Appendix ?, a rational \vartheta satisfying (?) can be computed in polynomial time. Therefore, the whole reduction runs in polynomial time.

We now show that the reduction is correct, that is, the instance (P_{XY}, R, \vartheta) of IB-Threshold is a yes-instance if and only if (G,k) is a yes-instance of Clique. Recall that (P_{XY}, R, \vartheta) is a yes-instance of IB-Threshold if there exists an encoder T of X such that I(X;T) \le R and I(Y;T) \ge \vartheta, and that (G,k) is a yes-instance of Clique if \omega(G)\geq k. Note that, by (?), we have that

Assume first that (G,k) is a yes-instance, that is, \omega(G) \ge k. We show that the encoder T_1 satisfies that I(X;T_1)\leq R and I(Y;T_1)\geq \vartheta, and therefore (P_{XY}, R, \vartheta) is a yes-instance. By Lemma 14, R^* = I(X;T_1), and in particular R^* \ge 0, since mutual information is non-negative. Since |\tilde R - R^*| \le \frac{\delta}{4}, we have that

In particular, since \tilde R + \frac{\delta}{4} \ge R^* \ge 0, the maximum in (?) equals \tilde R + \frac{\delta}{4}, that is, R = \tilde R + \frac{\delta}{4}. Therefore, from (?) we have that

The leftmost inequality in (?) gives I(X;T_1) = R^* \le R, so the encoder T_1 satisfies the rate constraint. Moreover, by Lemma 12 and R^*\geq R-\delta/2 from the rightmost inequality in (?), we get that

By the rightmost inequality in (?), (?) gives

Hence, T_1 satisfies both I(X;T_1) \le R and I(Y;T_1) \ge \vartheta, and (P_{XY}, R, \vartheta) is a yes-instance of IB-Threshold. Moreover, \mathrm{IB}(P_{XY}, R) \ge I(Y;T_1) > \vartheta + \frac{\delta}{4}.

Assume now that (G,k) is a no-instance, that is, \omega(G) < k. Since R \ge 0 by (?), Lemma 9 and the leftmost inequality in (?) give

Hence, every encoder T of X with I(X;T) \le R satisfies I(Y;T) \le \mathrm{IB}(P_{XY}, R) < \vartheta. Thus, (P_{XY}, R, \vartheta) is a no-instance of IB-Threshold.

Therefore, we have shown that the reduction is correct, and, since Clique is NP-hard, also when restricted to instances satisfying (?), IB-Threshold is NP-hard.

We now prove the inapproximability result. The previous argument shows that \mathrm{IB}(P_{XY}, R) > \vartheta + \frac{\delta}{4} if (G,k) is a yes-instance of Clique, and \mathrm{IB}(P_{XY}, R) \le \vartheta - \frac{\delta}{4} if (G,k) is a no-instance. Hence, any approximation of \mathrm{IB}(P_{XY}, R) within an additive error smaller than \frac{\delta}{4} lies on the same side of \vartheta as \mathrm{IB}(P_{XY}, R), and thus decides Clique. Suppose, for the sake of contradiction, that there exists a polynomial-time algorithm that, given P_{XY} and R, outputs a value v such that

By Lemma 13, \frac{1}{576 M^{3/2}} \le \frac{\delta}{8} < \frac{\delta}{4}. Hence, if \omega(G) \ge k, then v \ge \mathrm{IB}(P_{XY}, R) - \frac{\delta}{8} > \vartheta, while, if \omega(G) < k, then v \le \mathrm{IB}(P_{XY}, R) + \frac{\delta}{8} < \vartheta. Thus, running the reduction, then the algorithm, and comparing v with \vartheta would decide Clique in polynomial time, which is impossible unless \mathrm{P} = \mathrm{NP}.

For any y\in \mathcal{Y} and t\in \mathcal{T}, we have that

By definition of mutual information in (?) and of conditional entropy in (?), we have that

and similarly that

Since P_Y = W P_X and, by Lemma 1, P_{Y\mid T}(\cdot \mid t) = W P_{X\mid T}(\cdot \mid t) for every t \in \mathcal{T}, subtracting one half of (Proof) from (Proof) gives

which proves (2).

Finally, by (?) and (2), we obtain that

where the last equality holds because g_W(P_X) does not depend on T.

For every v \in V(G), by (?),

By (?), W_G(v \mid (w,u)) = 1/2 if v \in \{w,u\}, and 0 otherwise. If v \in C, exactly 2(j-1) oriented edges of A(C) have v as an endpoint, hence

If v \notin C, no oriented edge of A(C) has v as an endpoint, hence (W_G U_C)(v) = 0. This proves (3). Moreover, since H(W_G U_C) = \log j and H(U_C) = \log\big(j(j-1)\big), from (?) we finally get that

which concludes the proof.

Let \pi = W_G P, and let P_1 and P_2 denote the distributions on V(G) such that for any v \in V(G)

By (?), W_G(v\mid (w,u))=1/2 if and only if v\in\{w,u\}. Hence, we have that

Let us now extend P to a distribution on all V(G) \times V(G) by setting it to zero outside A(G). We observe that if P(u,v) > 0, then \pi(u) \ge \frac12 P_1(u) > 0 and \pi(v) \ge \frac12 P_2(v) > 0. Hence \operatorname{supp}(P) \subseteq \operatorname{supp}(\pi \otimes \pi), and D(P\|\pi\otimes\pi) is finite. Moreover, using (Proof), we obtain that

where the last equality follows from \pi= W_G P and (?).

Now, by applying Jensen’s inequality on the concavity of the logarithm we obtain the following lower bound on D(P\|\pi\otimes\pi):

By Lemma 4, (?) yields

Combining (Proof) and (?) yields (5). The equality for P = U_C instead follows from Lemma 3, since for a clique C of size \omega(G) we have that g_G(U_C)=c_{\omega(G)}.

By (?),

where the inequality follows from -\ln(1-x) \ge x for 0 \le x < 1.

1. By construction of {\bar G}, every edge of {\bar G} belongs to E(C_e) for some e\in E(G). Thus, it suffices to show that for each edge x\in E({\bar G}) there exists a unique e\in E(G) such that x\in E(C_e).

Let x=\{u,v\} be an edge of {\bar G}. Assume first that x\in E(G). Since for every e\in E(G) C_e\cap V(G)=e, we have that E(C_e)\cap E(G)=\{e\} for any e\in E(G). Hence, it follows that x\in E(C_e) if and only if x=e.

If x=\{u,v\}\notin E(G), then x is an edge added in the construction of {\bar G}, and at least one vertex between u and v, say u, belongs to S_e for some e \in E(G). Moreover, x\in E(C_e). Since the sets S_e are pairwise disjoint, it follows that u \notin C_{e'} for every e' \neq e. Hence, for every e'\neq e, we have that x \notin E(C_{e'}). Consequently, there exists a unique e\in E(G) such that x\in E(C_e).

Therefore, each edge of {\bar G} lies in exactly one of the sets E(C_e), with e \in E(G). Hence, \{E(C_e)\}_{e\in E(G)} forms a partition of E({\bar G}).

2. By construction of {\bar G}, for each edge e\in E(G), we add a clique of size m. Therefore, the total number of oriented edges is

3. We first show that \omega({\bar G}) \le \max\{\omega(G), m\}.

Let K be a clique of {\bar G}. If K contains a vertex z \in S_e for some e \in E(G), then, by construction of {\bar G}, all the neighbours of z, that are the vertices adjacent to z, belong to C_e. Therefore, it follows that K \subseteq C_e, and hence |K| \le |C_e| = m.

If K does not contain a vertex z\in S_e for any e\in E(G), then we have that K \subseteq V(G). Therefore, K is a clique of G, and |K| \le \omega(G).

Conversely, G is a subgraph of {\bar G}, so every clique of G is a clique of {\bar G}, and \omega({\bar G}) \ge \omega(G). Moreover, each C_e is a clique of {\bar G} of size m, and E(G) \neq \emptyset by assumption. Thus, \omega({\bar G}) \ge m. Putting all together we have that \omega({\bar G}) = \max\{\omega(G), m\}.

We first prove (8). For every e \in E(G), the set A(C_e) contains m(m-1) oriented edges, each of probability 1/M under P_X. Hence, by (?) we have that

where the last equality follows by recalling that M=m(m-1)N.

Moreover, by the Bayes’ rule, for every x \in A({\bar G}), we have that

that is, P_{X \mid T_0}(\cdot \mid e) = U_{C_e}, where U_{C_e} is the probability distribution on A({\bar G}) that is uniform on A(C_e) (cfr. (?)). This proves (8).

Now, we prove (8). By Lemma 2 and (8), it follows that

Since each C_e is a clique of {\bar G} of size m, Lemma 3 gives that g_{{\bar G}}(U_{C_e}) = c_m for every e \in E(G). This together with (?) proves (8). Finally, L_{1/2}(P_{XY}) \ge \Lambda_0 immediately follows from the definition of L_{1/2}(P_{XY}) in (?) as a supremum over all encoders.

Since \omega(G) \le k-1 = m, property 3 of Lemma 7 gives \omega({\bar G}) = m. Hence, by Lemma 5 applied to the graph {\bar G}, we have g_{{\bar G}}(Q) \ge c_m for every probability distribution Q on A({\bar G}). In particular, this holds for every posterior distribution P_{X\mid T}(\cdot \mid t) of an encoder T of X. Thus, by Lemma 2, for every encoder T of X we obtain that

which proves (9). Taking the supremum over all encoders gives L_{1/2}(P_{XY}) \le \Lambda_0, and by Lemma 8 we have that L_{1/2}(P_{XY})\geq \Lambda_0. Hence, L_{1/2}(P_{XY}) = \Lambda_0.

Now, if T is an encoder with I(X;T) \le R, then (9) gives

Taking the supremum over all such encoders yields (9).

Recall that M = N m(m-1) = N/\mu, so that P_X(x) = 1/M = \mu/N for every x \in A({\bar G}).

We first consider the case T_1=\star. By (?), we recall that P_{T_1 \mid X}(\star \mid x) = p if x \in A(K), and P_{T_1 \mid X}(\star \mid x) = 0 otherwise. Since K is a clique of size k, it has k(k-1) = 2B oriented edges. Therefore,

where the last equality follows from (?). Moreover, by the Bayes’ rule, for every x \in A(K), we have that

while P_{X \mid T_1}(x \mid \star) = 0 for x \notin A(K). By definition (?) of U_K, this means that P_{X \mid T_1}(\cdot \mid \star) = U_K.

We now consider T=e for e \in E(G). From (?) and (?), we recall that

Thus, T_1 differs from T_0 only on the oriented edges in A(C_e) \cap A(K). Let us determine this set. We claim that

To prove (?), let x = (u,v) \in A(K). Since K is a clique of G, the edge \{u,v\} is an edge of G. Moreover, x \in A(C_e) if and only if \{u,v\} \in E(C_e). By the proof of Lemma 7, the only edge of G that belongs to E(C_e) is e. Therefore, x \in A(C_e) if and only if \{u,v\} = e, that is, if and only if x is one of the two orientations of e. In other words, A(C_e) \cap A(K) consists of the orientations of e that belong to A(K). Therefore, if e \in E(K), both orientations of e belong to A(K), and hence A(C_e) \cap A(K) = A(e). If e \notin E(K), no orientation of e belongs to A(K), and hence A(C_e) \cap A(K) = \emptyset, proving (?).

Now, if e \in E(G) \setminus E(K), then A(C_e) \cap A(K) = \emptyset by (?). Therefore, (?) gives P_{T_1 \mid X}(e \mid x) = P_{T_0 \mid X}(e \mid x) for every x \in A({\bar G}), and, by Lemma 8 we get that P_{T_1}(e) = 1/N and P_{X \mid T_1}(\cdot \mid e) = U_{C_e}.

If e \in E(K), then A(C_e) \cap A(K) = A(e) by (?). Moreover, since |A(e)|=2 and C_e is a clique of size m, we have that |A(C_e) \setminus A(e)|=m(m-1) - 2. Hence, by (?), we obtain that

where the last equalities follow from (?). Moreover, by the Bayes’ rule, we have that

that is, P_{X \mid T_1}(\cdot \mid e) = V_e by (?).

The idea of the proof is to first show that g_{{\bar G}}(V_e) - c_m is at most one half of the relative entropy D(V_e \,\|\, U_{C_e}), then to bound this relative entropy by the \chi^2-divergence, and finally to compute the \chi^2-divergence explicitly.

We start by rewriting c_m in terms of \mu=1/(m(m-1)). By (?), we have that

Combining (Proof) with the definition (?) of g_{{\bar G}}, we obtain that

We bound the two terms on the right-hand side of (?) separately.

For the first term, note that \operatorname{supp}(V_e) \subseteq A(C_e) by (?), and that every oriented edge in A(C_e) has both endpoints in C_e. Hence, by (?), the distribution W_{{\bar G}} V_e is supported on C_e, which has m vertices. Consequently, H(W_{{\bar G}} V_e) \le \log m, and the first term \big(H(W_{{\bar G}} V_e) - \log m\big) in (?) is non-positive.

For the second term, we recall that U_{C_e}(x) = \mu for every x \in A(C_e). Then, since \operatorname{supp}(V_e) \subseteq A(C_e), we have that

Thus, by (?), the second term in (?) equals \frac12 D(V_e \,\|\, U_{C_e}), and we obtain that

Since \operatorname{supp}(V_e) \subseteq A(C_e) = \operatorname{supp}(U_{C_e}), we can apply Proposition 1 to bound the right-hand side of (?) yielding

Let us compute \chi^2(V_e \,\|\, U_{C_e}). By (?), we have that

From (?), we recall that V_e(x) = a_0 for the m(m-1) - 2 oriented edges x \in A(C_e) \setminus A(e), and V_e(x) = a_1 for the 2 oriented edges x \in A(e). Thus, using \mu p = \varepsilon/2, we have that

Substituting these values into (?), we obtain that

Recalling that \mu=1/(m(m-1)), we have that m(m-1)\mu = 1. Therefore, the expression in the parentheses in (?) simplifies to

Hence, we have that

Finally, combining (?), (?) and (?), we obtain that

which proves (11).

The idea of the proof is to compare the average cost of the posteriors of T_1 with the cost c_m of the posteriors of T_0. Indeed, the posterior U_K yields a gain that is linear in \varepsilon, while the perturbed posteriors V_e yield a loss that, by Lemma 11, is quadratic in \varepsilon. For our choice of \varepsilon in (?), the gain exceeds the loss by at least \delta.

Let

be the average cost of the posteriors of T_1. By Lemma 10, and since g_{{\bar G}}(U_K) = c_k and g_{{\bar G}}(U_{C_e}) = c_m by Lemma 3, we have that

Since the probabilities in (10) sum to 1, we can write c_m as

Subtracting (?) from \Psi, the terms corresponding to e \in E(G) \setminus E(K) cancel out, and we obtain that

The first term on the right-hand side of (?) is the gain due to U_K, and the second one is the loss due to the perturbed posteriors V_e. We bound the two terms separately.

For the gain, by Lemma 6, since k = m+1, we get that

For the loss, we first note that \varepsilon is small. Indeed, since m \ge 3, we have that \mu = \frac{1}{m(m-1)} \le \frac16 and m^2 \ge 9. Consequently, it follows from (?) that \varepsilon = \frac{\mu}{6m^2} \le \frac{1}{324}. Moreover, since 1 - \varepsilon > \frac12, and, since \ln 2 > \frac23, we have that

Recalling that |E(K)| = B, by Lemma 11 and (?) we have that

Recalling that \lambda = B\varepsilon/N and \varepsilon/\mu = 1/(6m^2), we obtain from (?) that

Substituting (?) and (?) into (?), we get that

Since \lambda = \frac{B\varepsilon}{N} = \frac{B\mu}{6Nm^2} and B\mu = \frac{k(k-1)}{2} \cdot \frac{1}{(k-1)(k-2)} = \frac{k}{2(k-2)} \ge \frac12 by (?), we can rewrite (?) as

where the last equality follows from (12).

Finally, by Lemma 2 and (?), we obtain that

which proves (12). The last statement L_{1/2}(P_{XY})> \Lambda_0 + \delta follows from the definition of L_{1/2}(P_{XY}) as a supremum over all encoders.

Let us start by expressing \delta in terms of M and m. By property 2 of Lemma 7, we have that M = N m(m-1), and consequently N = M/(m(m-1)). Substituting this into (12), we obtain

We now bound m from above in terms of M. Since k = m+1, we have that B = \frac{k(k-1)}{2} = \frac{m(m+1)}{2}, and by the assumption (?), it holds that N \ge B. Combining these facts, it follows that

Now, to remove the term -1 in (?), we note that m \ge 3 implies m^2 \ge 9, that is, 1 \le \frac{m^2}{9}. Hence m^2 - 1 \ge m^2 - \frac{m^2}{9} = \frac89 m^2. Therefore, we have that

which implies that

Finally, since m \ge 3, we have that 1 - \frac1m \ge \frac23. Using this and (?) in (Proof), we conclude that

We first prove (3). Let

One can see that \kappa_\beta, \rho_\beta and e_\beta are non-negative and satisfy \kappa_\beta+ \rho_\beta+e_\beta=1. Indeed, the equality \kappa_\beta + \rho_\beta + e_\beta = 1 holds by definition. Moreover, if \beta \le \frac12, then we have that \kappa_\beta = 2\beta, \rho_\beta = 0 and e_\beta = 1 - 2\beta \ge 0, while, if \beta \ge \frac12, then \kappa_\beta = 2(1-\beta), \rho_\beta = 2\beta - 1 \ge 0 and e_\beta = 0. Hence, (\kappa_\beta, \rho_\beta,e_\beta) is a probability distribution.

Let J be a random variable independent of (X,Y) taking values in \{0,1,2\} with probabilities \kappa_\beta, \rho_\beta and e_\beta, respectively, and let Z be the random variable on \mathcal{Z}= \big(\{0\} \times \mathcal{Y}\big) \cup \big(\{1\} \times \mathcal{X}\big) \cup \{\bot\}, where \bot is a new symbol, defined as

In other words, the random variable Z reveals Y with probability \kappa_\beta, reveals X with probability \rho_\beta, and reveals nothing with probability e_\beta.

Let \tilde P_{XZ} be the joint distribution of (X,Z). Since Z replaces Y while X is the same random variable as in (X,Y) \sim P_{XY}, the marginal distribution of X under \tilde P_{XZ} remains P_X. Consequently, since an encoder of X is a channel P_{T\mid X} from \mathcal{X} to \mathcal{T}, every encoder of X for P_{XY} is also an encoder of X for \tilde P_{XZ}, and conversely.

Let P_{T\mid X} be an encoder of X inducing T. Since J is independent of (X,Y,T) and it is a function of Z, by applying the chain rule we obtain that

where the last equality follows since I(J;T) = 0, and, given J, the variable Z is Y, X or a constant, with probabilities \kappa_\beta, \rho_\beta and e_\beta.

Now, subtracting \beta I(X;T) from both sides of (?), and using \beta - \rho_\beta = \kappa_\beta/2 by (Proof), we get that

Since the equality (?) holds for any encoder, taking the supremum over all encoders proves (3).

We now show that IB-Lagrangian_\beta is NP-hard by reducing IB-Lagrangian to it. Given an instance (P_{XY}, \tau) of IB-Lagrangian, consider the instance (\tilde P_{XZ}, \kappa_\beta \tau) of IB-Lagrangian_\beta, where Z is defined in (?). By (?), there exists an encoder T of X with I(Y;T) - \frac12 I(X;T) \ge \tau if and only if there exists an encoder T of X with I(Z;T) - \beta I(X;T) \ge \kappa_\beta \tau, that is, (P_{XY}, \tau) is a yes-instance of IB-Lagrangian if and only if (\tilde P_{XZ}, \kappa_\beta \tau) is a yes-instance of IB-Lagrangian_\beta. Therefore, to complete the reduction, it remains to show that the instance (\tilde P_{XZ}, \kappa_\beta \tau) can be computed in polynomial time. Since |\mathcal{Z}| = |\mathcal{X}| + |\mathcal{Y}| + 1, the table \tilde P_{XZ} has polynomially many entries. Moreover, since \beta is a fixed rational number, we have that \kappa_\beta, \rho_\beta and e_\beta are rational numbers. Hence, by (?), each entry of \tilde P_{XZ} is a fixed rational multiple of an entry of P_{XY} or of P_X. Thus, \tilde P_{XZ} and the rational threshold \kappa_\beta \tau can be computed in polynomial time. Therefore, by Theorem 1, IB-Lagrangian_\beta is NP-hard.

Finally, suppose that a polynomial-time algorithm approximates L_\beta within an additive error of \kappa_\beta / (216|\mathcal{X}|^{3/2}). Running it on \tilde P_{XZ}, which has the same input alphabet \mathcal{X} as P_{XY}, and dividing its output by \kappa_\beta, we obtain, by (3), an approximation of L_{1/2}(P_{XY}) within an additive error of 1/(216|\mathcal{X}|^{3/2}). By Theorem 1, this is impossible unless \mathrm{P} = \mathrm{NP}.

We first show that R^* depends only on N=|E(G)| and k. We recall that m = k-1, B = k(k-1)/2 and, by Lemma 7, M = N m(m-1). Moreover, by (?) and (?), we have that p = 1/(12m^2), \mu = 1/(m(m-1)), \varepsilon = 2\mu p and \lambda = B\varepsilon/N, and, by (?), a_0 = \mu/(1-\varepsilon) and a_1 = \mu(1-p)/(1-\varepsilon). Hence, all the quantities appearing in (14) depend only on N and k. We also note that a_0, a_1 > 0, since p < 1 and \varepsilon < 1, so that the logarithms in (14) are well defined.

Now assume that \omega(G) \ge k, so that the encoder T_1 defined in (?) exists. By definition of mutual information and conditional entropy in (?) and (?), we have that

We compute the two terms on the right-hand side of (?) separately. Since P_X is uniform on A({\bar G}), which has M elements, we have that H(P_X) = \log M. For the second term, recall from Lemma 10 that the posteriors of T_1 are:

• U_K, with probability \lambda;

• V_e for each e \in E(K), with probability (1-\varepsilon)/N;

• U_{C_e} for each e \in E(G) \setminus E(K), with probability 1/N.

The distribution U_K is uniform on the k(k-1) oriented edges of K, therefore it follows that H(U_K) = \log(k(k-1)). Similarly, each U_{C_e} is uniform on the m(m-1) oriented edges of C_e, and H(U_{C_e}) = \log(m(m-1)). While, by (?), V_e takes the value a_0 on the m(m-1) - 2 oriented edges of A(C_e) \setminus A(e), the value a_1 on the 2 oriented edges of A(e), and vanishes elsewhere. Hence, for every e \in E(K), it holds that

Putting these results together, and recalling that |E(K)| = B and |E(G) \setminus E(K)| = N - B, we obtain that

Substituting (?) and H(P_X) = \log M into (?) gives I(X;T_1) = R^*.