arXiv reader

On Minimum Divergence for Distribution Reconstruction

Justin Bec, Thomas Le Corre and Aurélien Garivier

arXiv
2610.21872v1 · PDF · 15 October 2026
Categories
math.ST, cs.IT, math.IT, stat.TH
Licence
CC BY 4.0 The site shows the text.

Abstract

We study the possibility of generalizing maximum-entropy reconstruction, which approximately recovers a probability distribution from finitely many (generalized) moments by minimizing the Kullback–Leibler (KL) divergence to a reference measure. The reconstruction lies in an exponential family and is also the best KL approximation of the unknown distribution within that family: moment matching and information projection coincide. Replacing KL by a smooth f-divergence, the reconstruction still exists in an associated family, but the best approximation property does not. Specifically, a family is called entropy-adapted if the projection of any distribution matches its moments. We show that, up to equivalence, KL is the only C^2 f-divergence that is entropy-adapted, whatever the choice of the generalized moments. We illustrate the consequences of this failure numerically.

Items 27

Remark

Remark 1

UN6P

When \mathcal{X} = [0,1] and T^{j}(x) = x^j these are the usual moments of \nu_0, which is where the terminology comes from. The moment functions are defined on the whole of \mathbb{R} because the same family \mathcal{T} will be used on several spaces \mathcal{X}, in particular on finite ones (Definition 8); nothing below depends on the structure of \mathbb{R}, which could be replaced by any infinite set.

Remark

Remark 2

MIIG

The picture is the one familiar from information geometry: the constraints defining M^{\mathcal{T}}_{\nu_0} are linear in the density, making it a mixture-flat (m-flat) affine set, whereas the log-densities of \mathcal{E}_{\mathrm{KL}} are affine in \mathcal{T}, making it an exponential-flat (e-flat) set. A one-parameter exponential family (k=1) is literally an e-geodesic; for general k it is an e-flat submanifold, and it is in this loose sense that we will speak of \mathcal{E}_{\mathrm{KL}} as the maximum-entropy “geodesic”.

Lemma

Lemma 3 (Pythagorean identity)

ANQ2

For every \nu_\theta \in \mathcal{E}_{\mathrm{KL}},

\mathrm{KL}(\nu_0, \nu_\theta) \;=\; \mathrm{KL}(\nu_0, \nu_{\bar\theta}) + \mathrm{KL}(\nu_{\bar\theta}, \nu_\theta)\,.
Remark

Remark 4

2IQ3

Concretely, (P1) is nothing but a finite-dimensional system of k scalar moment equations in the unknown \theta, namely \nu_\theta(\mathcal{T}) = \nu_0(\mathcal{T}). What could have been an optimization over an infinite-dimensional space of measures becomes the resolution of k scalar equations.

Definition

Definition 5

Y8VQ

A function f : \mathbb{R}^*_+ \to \mathbb{R} defines an f-divergence if it is strictly convex on \mathbb{R}^*_+ and f(1) = 0. The associated divergence is

D_f(\nu, \mu) = \begin{cases} \int_{\mathcal{X}} f\!\Bigl(\tfrac{\,\mathrm{d}\nu}{\,\mathrm{d}\mu}\Bigr) \,\mathrm{d}\mu, & \text{if } \nu \ll \mu, \\[4pt] +\infty, & \text{otherwise.} \end{cases}
Definition

Definition 6

OC9W

We will say that f is the generator of D_f, and that functions f, g are equivalent, written f \sim g, if they define the same f-divergence up to a multiplicative constant, that is if D_f is proportional to D_g.

Remark

Remark 7

NCZ4

Adding an affine term leads to an equivalent generator: for A>0 and B \in \mathbb{R}, if g(x) = Af(x) + B(x-1) then g \sim f.

Uses Definition 6 · Used by Proof, Notation
Definition

Definition 8 (\mathcal{T}-entropy-adapted)

F7MK

Let f be a C^2 f-divergence and \mathcal{T} = (T^{0}, \dots, T^{k}) a free family with T^{0} \equiv 1 and k \geq 1. We say that f is \mathcal{T}-entropy-adapted if, for every finite space \mathcal{X} = \{x_1,\dots,x_n\} \subset \mathbb{R} with reference probability measure \lambda of full support and every \nu \in \mathcal{P}(\mathcal{X}) equivalent to \lambda, if \mathcal{E}_f \cap M^{\mathcal{T}}_{\nu} is non-empty its (necessarily unique) element \nu_{\bar\theta} is such that

D_f(\mu, \nu_\theta) \;\ge\; D_f(\mu, \nu_{\bar\theta}) \qquad \text{for all } \mu \in M^{\mathcal{T}}_{\nu} \text{ and all } \nu_\theta \in \mathcal{E}_f\,.
Remark

Remark 9

MOFU

We restrict Definition 8 to finite spaces for two reasons. First, the proof of Theorem 10 only ever uses the hypothesis on finite spaces (Remark 12), so a weaker hypothesis makes the “only if” direction stronger. Second, on infinite spaces the moment matcher may fail to exist even for KL: take \mathcal{X} = [0,\infty), T(x) = x and \lambda(\,\mathrm{d} x) \propto e^{-x}(1+x^3)^{-1}\,\mathrm{d} x. Then e^{\theta x} is \lambda-integrable only for \theta \leq 1, so the means of the elements of \mathcal{E}_{\mathrm{KL}} are bounded by some m^* < \infty, while measures \nu \sim \lambda with finite mean larger than m^* exist. A definition quantifying over all spaces would therefore exclude KL itself. Finally, on a finite space all integrals are finite sums, and \mathcal{E}_f is simply parametrized by \{\theta \in \Theta_f : \sum_i \lambda_i (f')^{-1}(\theta^\top \mathcal{T}(x_i)) = 1\}. The definition could nevertheless be extended to infinite spaces under additional assumptions guaranteeing the existence of the moment matcher, for instance bounded moment functions \mathcal{T}, or more generally measures \nu whose moment vector lies in the image of the moment map of \mathcal{E}_f; Theorem 10 would remain valid, since finite spaces are still included.

Theorem

Theorem 10

BK91

Let \mathcal{T} = (T^{0}, \dots, T^{k}) be a free family with T^{0} \equiv 1 and k \geq 1, and let f be a C^2 f-divergence. Then, we have:

\text{$f$ is $\mathcal{T}$-entropy-adapted iff up to equivalence, } D_f = \mathrm{KL}\,.
Proof of Theorem 10

Fix s, u > 0; our goal is to prove that h'(s) = h'(u) when f''(1) \neq 0 (Lemma 11 then concludes), and that h'(u) = 0 when f''(1) = 0, taking s = 1 (which makes f affine and shows that this case in fact never occurs). Both conclusions will be read off Step 2 on a configuration built once and for all.

A preliminary reduction. Suppose first that some value V \in \mathbb{R}^k of the vector T is attained at least three times, and pick three distinct points x_1, x_2, x_3 \in T^{-1}(V). Since \mathcal{T} is free, T is not constant on \mathbb{R}, so there is x_4 \in T^{-1}(V') for some value V' \neq V; pick j with V'_j \neq V_j. On \mathcal{X} = \{x_1, \dots, x_4\}, every g \in S_\mathcal{X} takes the same value at x_1, x_2 and x_3. Lemma 14 (with coordinate j, pivots \{3, 4\} ordered according to the sign of V'_j - V_j, A = \{1, 2\} and B = \emptyset), or Remark 15 with weights (\epsilon, \epsilon, \frac12 - \epsilon, \frac12 - \epsilon) if f''(1) \neq 0, provides an anchored configuration on these four points in which the weights at x_1 and x_2 are small while \nu^\ast_3 stays bounded away from 0. Setting then r_1 = s, r_2 = u, r_4 = 1 and

r_3 = 1 - \frac{\nu^\ast_1\,(s-1) + \nu^\ast_2\,(u-1)}{\nu^\ast_3}\,,

positive for that choice of weights, the constraints of \mathcal{R}(\nu_\ast) hold by inspection (group the indices 1, 2, 3, where every g \in S_\mathcal{X} takes a common value). After the standing translation, \tau_j = 0 lies strictly between the two values V_j and V'_j taken by T^j on the configuration, so V_j \neq 0; and Step 2 at the indices 1 and 2, which share the value V hence the weight w_V, and at which every element of S_\mathcal{X} coincides, gives

w_V\, V_j\, h'(s) \;=\; w_V\, V_j\, h'(u)\,,

so h'(s) = h'(u). As s, u were arbitrary, h' is constant and Lemma 11 already proves the theorem in this situation (note that the argument used neither the value of f''(1) nor s = 1). From now on we may therefore assume that every value of T is attained at most twice. With the notation of Lemma 17, this forces d \ge 2: otherwise, by Lemma 17(c), T would be constant on the infinite set \mathbb{R} \setminus B. We fix j such that T^j is non-constant on \mathbb{R} \setminus B.

The common configuration. By Lemma 17(b), pick d points x_{d+1}, \dots, x_{2d} in \mathbb{R} \setminus B on which evaluation is a bijection from F' onto \mathbb{R}^d, and then d points x_1, \dots, x_d in \mathbb{R} \setminus (B \cup \{x_{d+1}, \dots, x_{2d}\}) with the same property. We write P = \{1, \dots, d\} and Q = \{d+1, \dots, 2d\} for the corresponding sets of indices. Since evaluation on Q is injective on F', T^j is non-constant on Q (otherwise T^j - c would vanish in F' for some constant c), and we pick \kappa, \Lambda \in Q with T^j_\kappa < T^j_\Lambda. Similarly, by Lemma 17(a), T^j is non-constant on \mathbb{R} \setminus (B \cup \{x_1, \dots, x_{2d}\}), so we can pick there two points y', y'' with T^j(y') \neq T^j(y''). The last point x_\ell, \ell = 2d + 1, will be one of them.

If f''(1) \neq 0, we take the trivial anchor \theta^\ast = 0 of Remark 15, with weights \lambda_i = \epsilon for i \in P \cup \{\ell\} and \lambda_i = (1 - (d+1)\epsilon)/d for i \in Q. If f''(1) = 0, we apply Lemma 14 to the points x_1, \dots, x_{2d}, y', y'', with coordinate j and pivots (\kappa, \Lambda); we then fix \eta \in (0, \delta] and use the weights of Lemma 14 with A = P \cup \{\ell\} and B = Q \setminus \{\kappa, \Lambda\}. In both cases \theta^\ast, hence every p_i and w_i, depends neither on \epsilon nor on the choice of x_\ell, and as \epsilon \to 0 the weights \lambda_i, i \in Q, converge to positive limits \lambda^0_i. Consequently \nu^\ast_i \to \nu^0_i \mathrel{\overset{\text{def}}{=}} \lambda^0_i p_i > 0 for i \in Q, \nu^\ast_i \to 0 for i \in P \cup \{\ell\}, and the barycentre (before translation) converges:

\tau_j \;\longrightarrow\; \tau^0_j \mathrel{\overset{\text{def}}{=}} \frac{\sum_{i \in Q} \nu^0_i\, w_i\, T^j_i}{\sum_{i \in Q} \nu^0_i\, w_i}\,,

a number which does not depend on x_\ell. We choose x_\ell \in \{y', y''\} with T^j(x_\ell) \neq \tau^0_j and set \mathcal{X} = \{x_1, \dots, x_{2d}, x_\ell\}, so that n = 2d + 1. For \epsilon small enough, these data form an anchored configuration on \mathcal{X}, and T^j(x_\ell) \neq \tau_j; after the standing translation, T^j_\ell \neq 0.

Since \mathcal{X} \subset \mathbb{R} \setminus B and evaluation on Q is injective on F', \dim S_\mathcal{X} = d and the restriction maps S_\mathcal{X} \to \mathbb{R}^P and S_\mathcal{X} \to \mathbb{R}^Q are bijective. Prescribe

r_i = s \quad (i \in P)\,, \qquad r_\ell = u\,,

and let g^1, \dots, g^d be a basis of S_\mathcal{X}. The condition r \in \mathcal{A}(\nu_\ast) is the linear system

\sum_{i \in Q} \nu^\ast_i\, g^m_i\, (r_i - 1) \;=\; - \sum_{i \in P \cup \{\ell\}} \nu^\ast_i\, g^m_i\, (r_i - 1) \qquad (m = 1, \dots, d)

in the unknowns (r_i - 1)_{i \in Q}. As \epsilon \to 0, its matrix converges to \bigl(\nu^0_i\, g^m_i\bigr)_{m,\, i \in Q}, which is invertible since (g^m_i)_{m,\, i \in Q} is, while its right-hand side is O(\epsilon). Hence the solution exists for \epsilon small and satisfies r_i \to 1 for i \in Q; in particular r_i > 0 and r \in \mathcal{R}(\nu_\ast). We fix such an \epsilon; by Step 2, \zeta^j(r) \in S_\mathcal{X}. We can now conclude.

Case 1: f''(1) = 0.

We show that this case cannot occur. Here h'(1) = -f''(1) = 0; take s = 1 in the construction above. At the indices i \in P, where r_i = 1, the vector \zeta^j(r) = \bigl(w_i\, T^j_i\, h'(r_i)\bigr)_i vanishes; as S_\mathcal{X} \to \mathbb{R}^P is injective, \zeta^j(r) = 0. At the index \ell this reads

w_\ell\, T^j_\ell\, h'(u) = 0\,,

and w_\ell > 0, T^j_\ell \neq 0 give h'(u) = 0. As u > 0 was arbitrary, h' \equiv 0 on \mathbb{R}^*_+, so f''(x) = -h'(x)/x vanishes identically and f is affine (contradicting strict convexity). A \mathcal{T}-entropy-adapted C^2 divergence therefore satisfies f''(1) \neq 0, and Case 2 proves the theorem.

Case 2: f''(1) \neq 0.

The anchor is the trivial one of Remark 15: \nu_\ast = \lambda and the weights are constant, w_i \equiv w = 1/f''(1) > 0. Since T^j|_\mathcal{X} \in S_\mathcal{X}, the vector

\zeta^j(r) - w\, h'(s)\, T^j|_\mathcal{X} \;=\; \bigl(w\, T^j_i\, (h'(r_i) - h'(s))\bigr)_{1 \le i \le n}

belongs to S_\mathcal{X} and vanishes on P, where r_i = s; it is therefore zero. At the index \ell this gives

w\, T^j_\ell\, \bigl(h'(u) - h'(s)\bigr) = 0\,,

and T^j_\ell \neq 0 gives h'(u) = h'(s). As s, u > 0 were arbitrary, h' is constant on \mathbb{R}^*_+ and Lemma 11 concludes: up to equivalence, D_f = \mathrm{KL}.

Lemma

Lemma 11

BMKN

Let f be a C^2 f-divergence and set h(x) = f(x) - x f'(x), so that h'(x) = -x f''(x). If h' is constant on \mathbb{R}^*_+ (equivalently, if x \mapsto x f''(x) is constant) then, up to equivalence, D_f = \mathrm{KL}.

Used by Proof
Proof of Lemma 11

Write h'(x) = -x f''(x) = -A for all x > 0. If A = 0 then f'' \equiv 0 and f is affine, contradicting strict convexity; convexity of f then forces A > 0. Integrating f''(x) = A/x twice, there exist C, D \in \mathbb{R} such that

f(x) = A\, x \ln x - A x + C x + D\,,

and the normalization f(1) = 0 gives D = A - C, that is

f(x) = A\, x \ln x + (C - A)(x - 1)\,.

By Remark 7, f \sim x \mapsto x \ln x, whose divergence is exactly \mathrm{KL}.

Remark

Remark 12

Z8DP

By Definition 8, the hypothesis only concerns finite spaces \mathcal{X} = \{x_1, \dots, x_n\} \subset \mathbb{R} (distinct points). Reference measures are then vectors: we write \lambda = \sum_{i=1}^n \lambda_i \delta_{x_i} with \lambda_i > 0 and \sum_i \lambda_i = 1, and T^j_i \mathrel{\overset{\text{def}}{=}} T^j(x_i), thereby identifying functions on \mathcal{X} with vectors of \mathbb{R}^n. We set

S_\mathcal{X} \;\mathrel{\overset{\text{def}}{=}}\; \operatorname{span}\bigl(1, T^1|_\mathcal{X}, \dots, T^k|_\mathcal{X}\bigr) \;\subset\; \mathbb{R}^n\,.

Since T^0 \equiv 1, a probability measure \mu with full support on \mathcal{X} belongs to M^{\mathcal{T}}_{\nu} if and only if \sum_i \mu_i g_i = \sum_i \nu_i g_i for every g \in S_\mathcal{X}: the moment sets depend on \mathcal{T} only through S_\mathcal{X}, and the family \mathcal{T} need not be free on \mathcal{X}. Every integral is a finite sum, so all the differentiations under the integral sign performed below are automatic.

Uses Definition 8 · Used by Remark 9
Remark

Remark 13

B52T

Let c = (c_1, \dots, c_k) \in \mathbb{R}^k. Translating each T^j by the constant c_j changes neither the moment sets (all measures involved being probability measures) nor the family \mathcal{E}_f, since \operatorname{span}(1, T^1 - c_1, \dots, T^k - c_k) = \operatorname{span}(1, T^1, \dots, T^k). Hence, if f is \mathcal{T}-entropy-adapted, it is also entropy-adapted for the translated family, for every c \in \mathbb{R}^k. We will use this freedom with c = \tau, a barycentre attached to the configurations constructed below.

Notation

Notation

J5SV

From now on we normalize f'(1) = 0, which by Remark 7 changes neither D_f, nor \mathcal{E}_f, nor f''. We write

Z \mathrel{\overset{\text{def}}{=}} \{x > 0 : f''(x) = 0\}\,.

By strict convexity, Z is closed with empty interior (on any interval where f'' \equiv 0 the function f would be affine). The derivative f' is continuous and strictly increasing, hence a homeomorphism from \mathbb{R}^*_+ onto the open interval I \mathrel{\overset{\text{def}}{=}} f'(\mathbb{R}^*_+), which contains 0 = f'(1); consequently f'(Z) is closed in I, and the inverse (f')^{-1} : I \to \mathbb{R}^*_+ is continuous, increasing, and C^1 on the open set I \setminus f'(Z), with

\bigl((f')^{-1}\bigr)' \;=\; \frac{1}{f'' \circ (f')^{-1}}

there, by the inverse function theorem. Finally, by Sard’s theorem in dimension one(Elementary in dimension one: fix a compact interval [a,b] \subset \mathbb{R}^*_+ and \varepsilon > 0. By uniform continuity of f'' on [a,b], there is \delta > 0 such that |f''| \le \varepsilon at every point of [a,b] at distance \le \delta from Z (recall f'' = 0 on Z). Split [a,b] into at most (b-a)/\delta + 1 intervals of length \le \delta: each one meeting Z is made of such points, so f' has oscillation \le \varepsilon\delta on it, and f'(Z \cap [a,b]) has outer measure at most \varepsilon(b - a + \delta). Let \varepsilon \to 0, then exhaust \mathbb{R}^*_+ by compact intervals.), the set f'(Z) of critical values of f' is Lebesgue-negligible. Note right away that when f''(1) \neq 0 none of this machinery will be needed: it only serves to handle the degenerate case f''(1) = 0.

Lemma

Lemma 14 (Existence of a non-degenerate anchor)

9CM2

Let y_1, \dots, y_N \in \mathbb{R} be distinct points, j \in \{1, \dots, k\}, and fix two indices \kappa, \Lambda with T^j(y_\kappa) < T^j(y_\Lambda). There exist \theta^\ast_0 \in \mathbb{R} and \theta^\ast_j > 0 such that, setting all the other coordinates of \theta^\ast to zero, m_i \mathrel{\overset{\text{def}}{=}} \theta^\ast_0 + \theta^\ast_j T^j(y_i) and p_i \mathrel{\overset{\text{def}}{=}} (f')^{-1}(m_i):

  • (i) m_i \in I \setminus f'(Z) for every i \in \{1, \dots, N\}; in particular f''(p_i) > 0, and \theta \mapsto p_{\theta,i} \mathrel{\overset{\text{def}}{=}} (f')^{-1}\bigl(\theta_0 + \sum_l \theta_l T^l(y_i)\bigr) is C^1 in a neighbourhood of \theta^\ast;

  • (ii) p_\kappa < 1 < p_\Lambda.

Moreover, let \mathcal{X} be a subset of \{y_1, \dots, y_N\} containing y_\kappa and y_\Lambda, and split its other indices into two disjoint sets A and B. There exists \delta > 0 such that, for all (\epsilon, \eta) \in [0, \delta]^2, the solution (\lambda_\kappa, \lambda_\Lambda) of

\lambda_\kappa + \lambda_\Lambda = 1 - \epsilon\, |A| - \eta\, |B|\,, \qquad \lambda_\kappa\, p_\kappa + \lambda_\Lambda\, p_\Lambda = 1 - \epsilon \sum_{i \in A} p_i - \eta \sum_{i \in B} p_i

has positive coordinates and depends continuously on (\epsilon, \eta). In particular, for (\epsilon, \eta) \in (0, \delta]^2, the weights \lambda_i = \epsilon (i \in A), \lambda_i = \eta (i \in B), \lambda_\kappa, \lambda_\Lambda, together with \theta^\ast, turn \mathcal{X} into an anchored configuration.

Uses Notation · Used by Proof
Proof of Lemma 14

Write \theta_0, \theta_1 for \theta^\ast_0, \theta^\ast_j, and T_i for T^j(y_i). Fix \theta_1 > 0, to be shrunk below. Since (f')^{-1} is increasing with (f')^{-1}(0) = 1, condition (ii) reads m_\kappa < 0 < m_\Lambda, i.e.

\theta_0 \in J \mathrel{\overset{\text{def}}{=}} \bigl(-\theta_1 T_\Lambda,\, -\theta_1 T_\kappa\bigr)\,,

a non-empty open interval, of length \theta_1 (T_\Lambda - T_\kappa). For \theta_0 \in J one has |\theta_0| \le \theta_1 \max(|T_\kappa|, |T_\Lambda|), hence

\max_i |m_i| \;\le\; \theta_1 \Bigl( \max(|T_\kappa|, |T_\Lambda|) + \max_i |T_i| \Bigr)\,;

since I is open and contains 0, taking \theta_1 small enough guarantees m_i \in I for every i and every \theta_0 \in J. With \theta_1 now fixed, the failure of the first part of (i) means \theta_0 \in \bigcup_i \bigl(f'(Z) - \theta_1 T_i\bigr), a finite union of Lebesgue-null sets, hence a null set; since J has positive measure, we may pick \theta_0 \in J outside it, so that m_i \notin f'(Z) for every i. This proves (i)–(ii): the strict positivity f''(p_i) > 0 follows from f'' \ge 0 (convexity) together with p_i \notin Z, and the C^1 regularity of \theta \mapsto p_{\theta,i} near \theta^\ast from the inverse function theorem at the non-critical point m_i.

For the weights, the linear system in (\lambda_\kappa, \lambda_\Lambda) has determinant p_\Lambda - p_\kappa > 0, independent of (\epsilon, \eta), so its solution is continuous (indeed affine) in (\epsilon, \eta). At (\epsilon, \eta) = (0, 0) it equals

(\lambda_\kappa, \lambda_\Lambda) \;=\; \Bigl( \frac{p_\Lambda - 1}{p_\Lambda - p_\kappa},\; \frac{1 - p_\kappa}{p_\Lambda - p_\kappa} \Bigr)\,,

whose coordinates are positive by (ii); by continuity, they remain positive on [0, \delta]^2 for some \delta > 0. The two equations are exactly \sum_i \lambda_i = 1 and \sum_i \lambda_i\, p_i = 1, which, together with (i), is the definition of an anchored configuration.

Remark

Remark 15

2342

When f''(1) \neq 0, no lemma is needed: the choice \theta^\ast = 0 turns any finite configuration with any positive weights into an anchored configuration. Indeed m_i = 0 for every i, so p_i = (f')^{-1}(0) = 1, f''(p_i) = f''(1) \neq 0, and the normalization \sum_i \lambda_i p_i = \sum_i \lambda_i = 1 is automatic; the anchor is then simply \nu_\ast = \lambda itself, with \nu^\ast_i = \lambda_i.

Used by Proof
Step

Step 1 (Anchored Lagrange identity)

I791

For every anchored configuration, every r \in \mathcal{R}(\nu_\ast) and every j \in \{1, \dots, k\},

\sum_{i=1}^n \nu^\ast_i\, w_i\, T^j_i\, h(r_i) \;=\; 0\,.
Used by Proof
Proof of Step 1

The measure \nu_\ast belongs to \mathcal{E}_f \cap M^{\mathcal{T}}_{\nu_\ast}, so the uniqueness part of Definition 8, applied with \nu = \nu_\ast, gives \nu_{\bar\theta} = \nu_\ast: the moment matcher of \nu_\ast is \nu_\ast itself. The adaptedness inequality then yields, for \mu \mathrel{\overset{\text{def}}{=}} r\nu_\ast \in M^{\mathcal{T}}_{\nu_\ast},

D_f(\mu, \nu_\theta) \;\ge\; D_f(\mu, \nu_\ast) \qquad \text{for all } \nu_\theta \in \mathcal{E}_f\,.

The density of \mu with respect to \lambda being r p_\ast, this says that \theta^\ast minimizes

\theta \;\longmapsto\; \sum_i \lambda_i\, f\!\Bigl(\frac{r_i\, p_i}{p_{\theta,i}}\Bigr)\, p_{\theta,i} \qquad \text{subject to} \qquad \sum_i \lambda_i\, p_{\theta,i} = 1\,,

with p_{\theta,i} = (f')^{-1}\bigl(\theta_0 + \sum_l \theta_l T^l_i\bigr), over a neighbourhood of \theta^\ast in \mathbb{R}^{k+1} on which every \theta_0 + \sum_l \theta_l T^l_i stays in the open set I \setminus f'(Z). Since f''(p_i) > 0 and f'' is continuous, both the objective and the constraint are C^1 near \theta^\ast, with

\frac{\partial p_{\theta,i}}{\partial \theta_l} = \frac{T^l_i}{f''(p_{\theta,i})} \qquad (l = 0, \dots, k,\ T^0 \equiv 1)\,.

Moreover, for any fixed q > 0,

\frac{\,\mathrm{d}}{\,\mathrm{d} p}\Bigl[ f\!\Bigl(\frac{q}{p}\Bigr)\, p \Bigr] = f'\!\Bigl(\frac{q}{p}\Bigr)\Bigl(-\frac{q}{p^2}\Bigr)\, p + f\!\Bigl(\frac{q}{p}\Bigr) = f\!\Bigl(\frac{q}{p}\Bigr) - \frac{q}{p}\, f'\!\Bigl(\frac{q}{p}\Bigr) = h\!\Bigl(\frac{q}{p}\Bigr)\,.

Applying this with q = r_i p_i and evaluating at \theta = \theta^\ast, where p_{\theta^\ast, i} = p_i (so that q/p = r_i), the partial derivatives in \theta_l of the objective and of the constraint at \theta^\ast are

\sum_i \lambda_i\, T^l_i\, \frac{h(r_i)}{f''(p_i)} \qquad \text{and} \qquad \sum_i \lambda_i\, T^l_i\, \frac{1}{f''(p_i)}\,, \qquad l = 0, \dots, k\,.

The constraint gradient is non-zero (its \theta_0-coordinate is \sum_i \lambda_i / f''(p_i) > 0) so the Lagrange multiplier rule applies: there exists \alpha \in \mathbb{R} such that, for l = 0, \dots, k,

\sum_i \lambda_i\, T^l_i\, \frac{h(r_i)}{f''(p_i)} = \alpha \sum_i \lambda_i\, T^l_i\, \frac{1}{f''(p_i)}\,.

Since \lambda_i / f''(p_i) = \lambda_i p_i \cdot \frac{1}{p_i f''(p_i)} = \nu^\ast_i w_i, the equation with index l = j \geq 1 reads

\sum_i \nu^\ast_i\, w_i\, T^j_i\, h(r_i) \;=\; \alpha \sum_i \nu^\ast_i\, w_i\, T^j_i \;=\; \alpha\, W \tau_j \;=\; 0\,,

by the standing convention \tau = 0. (The equation with index l = 0 merely determines \alpha, which we do not need.)

Step

Step 2 (Anchored affine relation)

1KOB

For every anchored configuration, every r \in \mathcal{R}(\nu_\ast) and every j \in \{1, \dots, k\},

\zeta^j(r) \;\mathrel{\overset{\text{def}}{=}}\; \bigl(w_i\, T^j_i\, h'(r_i)\bigr)_{1 \le i \le n} \;\in\; S_\mathcal{X}\,,

that is, there exist a_r, b_{r,1}, \dots, b_{r,k} \in \mathbb{R} (depending on j) such that

\forall i \in \{1, \dots, n\}, \qquad w_i\, T^j_i\, h'(r_i) \;=\; a_r + \sum_{l=1}^k b_{r,l}\, T^l_i\,.
Used by Proof
Proof of Step 2

Let r \in \mathcal{R}(\nu_\ast) and v \in \mathcal{V}(\nu_\ast). Since \mathcal{R}(\nu_\ast) is open in \mathcal{A}(\nu_\ast), there is t_0 > 0 such that r + tv \in \mathcal{R}(\nu_\ast) for all t \in (-t_0, t_0), and identity (1) (whose coefficients \nu^\ast_i w_i T^j_i do not depend on r) holds at every r + tv. Differentiating in t at t = 0 (licit, h being C^1 since f is C^2),

\sum_i \nu^\ast_i\, w_i\, T^j_i\, v_i\, h'(r_i) = 0 \qquad \text{for all } v \in \mathcal{V}(\nu_\ast)\,.

This says that \zeta^j(r) is \langle \cdot, \cdot \rangle_{\nu^\ast}-orthogonal to \mathcal{V}(\nu_\ast) = S_\mathcal{X}^{\perp_{\nu^\ast}}, hence \zeta^j(r) \in \bigl(S_\mathcal{X}^{\perp_{\nu^\ast}}\bigr)^{\perp_{\nu^\ast}} = S_\mathcal{X}. This concludes.

Remark

Remark 16

PCUJ

As a sanity check, for the KL generator f(x) = x \ln x - x + 1 (which satisfies f'(1) = 0) one has h' \equiv -1 and w_i = 1/(p_i \cdot \tfrac{1}{p_i}) = 1, so that \zeta^j(r) = -T^j|_\mathcal{X} \in S_\mathcal{X} whatever r: as it must (KL being \mathcal{T}-entropy-adapted) Step 2 carries no information in that case. It is the possible non-constancy of h' for other generators that the relation will now be played against. For k = 1, Step 2 reads w_i T_i h'(r_i) = a_r + b_r T_i.

Lemma

Lemma 17 (Choice of points)

870U

Let F = \operatorname{span}(1, T^1, \dots, T^k), viewed as a space of functions on \mathbb{R}, let F_0 \subset F be the subspace of its finitely supported elements, and let B be the union of the supports of the elements of F_0. Then B is finite. Let F' be the space of restrictions to \mathbb{R} \setminus B of the elements of F, and d \mathrel{\overset{\text{def}}{=}} \dim F' \ge 1. Then:

  • (a) for every finite C \subset \mathbb{R} \setminus B, the restriction map from F' to functions on \mathbb{R} \setminus (B \cup C) is injective;

  • (b) for every finite C \subset \mathbb{R} \setminus B, there exist d points y_1, \dots, y_d \in \mathbb{R} \setminus (B \cup C) such that g \mapsto (g(y_1), \dots, g(y_d)) is a bijection from F' onto \mathbb{R}^d;

  • (c) d = 1 if and only if every T^j is constant on \mathbb{R} \setminus B.

Uses Notation · Used by Proof
Proof of Lemma 17

F_0 is finite-dimensional; if g^1, \dots, g^m is a basis, every element of F_0 is supported in \bigcup_l \mathrm{supp}\, g^l, which is finite, so B is finite. Since \mathbb{R} \setminus B is non-empty, the constant 1 has a non-zero restriction, so d \ge 1. (a) Let g \in F vanish on \mathbb{R} \setminus (B \cup C). Then g is supported in the finite set B \cup C, so g \in F_0 and g vanishes outside B: its restriction to \mathbb{R} \setminus B is zero. (b) By (a), the restrictions of the elements of F' to E \mathrel{\overset{\text{def}}{=}} \mathbb{R} \setminus (B \cup C) form a space F'' of dimension d. The evaluations g \mapsto g(y), y \in E, span the dual of F'': otherwise some non-zero g \in F'' would vanish at every point of E. Extract a basis y_1, \dots, y_d of evaluations; the corresponding map is a bijection F'' \to \mathbb{R}^d, hence, by (a), a bijection F' \to \mathbb{R}^d. (c) F' contains the constants, and is spanned by them and the restrictions of the T^j.

Remark

Remark 18

38ID

For k = 1 and a function T taking each value at most twice, one has B = \emptyset and d = 2, and the common configuration consists of five points: P = \{1, 2\} carries the prescribed value s, Q = \{3, 4\} consists of the two pivots used to balance the constraints, and x_5 is the test point. No reduction to families of the form (1, T) is needed: such a reduction would require that a \mathcal{T}-entropy-adapted f be (1, T^j)-entropy-adapted, which does not follow directly from Definition 8.

Source and licence

The text on this page is from arXiv:2610.21872v1 by Justin Bec, Thomas Le Corre and Aurélien Garivier, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2610/2610.21872.jsonl.