Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization

Hoi-To Wai, Zhuoran Yang, Zhaoran Wang and Mingyi Hong

arXiv: 1806.00877v4 · PDF · 9 January 2019
Categories: cs.LG, math.OC, stat.ML
Licence: CC BY 4.0 The site shows the text.

Abstract

Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents. Motivated by decentralized applications such as sensor networks, swarm robotics, and power grids, we study policy evaluation in MARL, where agents with jointly observed state-action pairs and private local rewards collaborate to learn the value of a given policy. In this paper, we propose a double averaging scheme, where each agent iteratively performs averaging over both space and time to incorporate neighboring gradient information and local reward information, respectively. We prove that the proposed algorithm converges to the optimal solution at a global geometric rate. In particular, such an algorithm is built upon a primal-dual reformulation of the mean squared projected Bellman error minimization problem, which gives rise to a decentralized convex-concave saddle-point problem. To the best of our knowledge, the proposed double averaging primal-dual optimization algorithm is the first to achieve fast finite-time convergence on decentralized convex-concave saddle-point problems.

Items 13

Notation

WBPL

Unless otherwise specified, for a vector {\bm x}, \| {\bm x} \| denotes its Euclidean norm; for a matrix {\bm X}, \| {\bm X} \| denotes its spectral norm, i.e., the largest singular value.

Used by Lemma 1, Theorem 1, Theorem, Notation, Lemma 2

Lemma

Lemma 1

KCBP

For all t \geq 1 and consider Algorithm ?, it holds that

\textstyle {\bm g}_{\bm{\theta}}(t) = \frac{1}{NM} \sum_{i=1}^N \sum_{p=1}^M {\nabla}_{\bm{\theta}} J_{i,p} ( \bm{\theta}_i^{\tau_p^t}, {\bm w}_i^{\tau_p^t} ) \;.\vspace{-.2cm}

Uses Notation · Used by Proof Sketch, Notation

Proof of Lemma 1

We shall prove the statement using induction. For the base case with t = 1, using (?) and the update rule specified in the algorithm, we have

{\bm g}_{\bm{\theta}}(1) = \frac{1}{N} \sum_{i=1}^N \frac{1}{M} {\nabla}_{\bm{\theta}} J_{i,p_1} ( \bm{\theta}_i^1, {\bm w}_i^1 ) = \frac{1}{NM} \sum_{i=1}^N \sum_{p=1}^M {\nabla}_{\bm{\theta}} J_{i,p_t} ( \bm{\theta}_i^{\tau_p^1}, {\bm w}_i^{\tau_p^1} ) \;,\vspace{-.1cm}

where we use the fact {\nabla}_{\bm{\theta}} J_{i,p} ( \bm{\theta}_i^{\tau_p^1}, {\bm w}_i^{\tau_p^1} ) = {\nabla}_{\bm{\theta}} J_{i,p} ( \bm{\theta}_i^{0}, {\bm w}_i^{0} ) = {\bm 0} for all p \neq p_1 in the above. For the induction step, suppose (1) holds up to iteration t. Since {\bm W} is doubly stochastic, (?) implies

\begin{split} {\bm g}_{\bm{\theta}}(t+1) & = \frac{1}{N} \sum_{i=1}^N \bigg \{ \sum_{j=1}^N W_{ij} {\bm s}_j^t + \frac{1}{M} \Big[ {\nabla}_{\bm{\theta}} J_{i,p_{t+1}} ( \bm{\theta}_i^{t+1}, {\bm w}_i^{t+1} ) - {\nabla}_{\bm{\theta}} J_{i,p_{t+1}} ( \bm{\theta}_i^{\tau_{p_{t+1}}^{t}}, {\bm w}_i^{\tau_{p_{t+1}}^{t}} ) \Big] \bigg\} \\ & = {\bm g}_{\bm{\theta}}(t) + \frac{1}{NM} \sum_{i=1}^N \Big[ {\nabla}_{\bm{\theta}} J_{i,p_{t+1}} ( \bm{\theta}_i^{t+1}, {\bm w}_i^{t+1} ) - {\nabla}_{\bm{\theta}} J_{i,p_{t+1}} ( \bm{\theta}_i^{\tau_{p_{t+1}}^{t}}, {\bm w}_i^{\tau_{p_{t+1}}^{t}} ) \Big]\;.\\[-.4cm] \end{split}\vspace{-.4cm}

Notice that we have \tau_{p_{t+1}}^{t+1} = t+1 and \tau_{p}^{t+1} = \tau_{p}^{t} for all p \neq p_{t+1}. The induction assumption in (1) can be written as

\begin{aligned} {\bm g}_{\bm{\theta}}(t) = \frac{1}{NM} \sum_{i=1}^N \biggl [ \sum_{p\neq p_{t+1} } {\nabla}_{\bm{\theta}} J_{i,p} ( \bm{\theta}_i^{\tau_p^{t+1}}, {\bm w}_i^{\tau_p^{t+1}} ) \biggr ] + \frac{1}{NM}\sum_{i=1}^N {\nabla}_{\bm{\theta}} J_{i,p_{t+1} } ( \bm{\theta}_i^{\tau_{p_{t+1}} ^{t}}, {\bm w}_i^{\tau_{p_{t+1}} ^{t } } ) \;. \end{aligned}

Finally, combining (Proof) and (Proof), we obtain the desired result that (1) holds for the t+1th iteration. This, together with (Proof),

establishes Lemma 1. Q.E.D.

As for the dual update (?), we observe the variable {\bm w}_i is local to agent i. Therefore its gradient surrogate, {\bm d}_i^t, involves only the tracking step over time [cf. (?)], i.e., it only averages the gradient over samples. Combining with Lemma 1 shows that the PD-DistIAG method uses gradient surrogates that are averages over samples despite the disparities across agents. Since the average over samples are done in a similar spirit as the SAG method, the proposed method is expected to converge linearly.

Storage and Computation Complexities Let us comment on the computational and storage complexity of PD-DistIAG method. First of all, since the method requires accessing the previously evaluated gradients, each agent has to store 2M such vectors in the memory to avoid re-evaluating these gradients. Each agent needs to store a total of 2Md real numbers. On the other hand, the per-iteration computation complexity for each agent is only {\cal O}(d) as each iteration only requires to evaluate the gradient over one sample, as delineated in (?)–(?).

Communication Overhead The PD-DistIAG method described in Algorithm ? requires an information exchange round [of {\bm s}_i^t and \bm{\theta}_i^t] among the agents at every iteration. From an implementation stand point, this may incur significant communication overhead when d \gg 1, and it is especially ineffective when the progress made in successive updates of the algorithm is not significant. A natural remedy is to perform multiple local updates at the agent using different samples without exchanging information with the neighbors. In this way, the communication overhead can be reduced. Actually, this modification to the PD-DistIAG method can be generally described using a time varying weight matrix {\bm W}(t), such that we have {\bm W}(t) = {\bm I} for most of the iteration. The convergence of PD-DistIAG method in this scenario is part of the future work.

A 1

HU6I

A sample is selected at least once for every M iterations, | t - \tau_p^t | \leq M for all p \in [M], t \geq 1.

A 2

9G7E

The sampled correlation matrix \hat{\bm A} is full rank, and the sampled covariance \hat{\bm C} is non-singular.

Theorem

Theorem 1

AQSP

Under A1 and A2, we denote by ( \bm{\theta}^\star, \{ {\bm w}_i^\star \}_{i=1}^N ) the primal-dual optimal solution to the optimization problem in (?). Set the step sizes as \gamma_2 = \beta \gamma_1 with \beta \mathrel{\mathop:}= 8 ( \rho + \lambda_{\sf max} ( \hat{\bm A}^\top \hat{\bm C}^{-1} \hat{\bm A}) ) / \lambda_{\sf min}( \hat{\bm C}). Define \overline{\bm{\theta}}(t) \mathrel{\mathop:}= \frac{1}{N} \sum_{i=1}^N \bm{\theta}_i^t as the average of parameters. If the primal step size \gamma_1 is sufficiently small, then there exists a constant 0 < \sigma < 1 that

\textstyle \big\| \overline{\bm{\theta}}(t) - \bm{\theta}^\star \big\|^2 + (1/\beta N) \sum_{i=1}^N \big\| {\bm w}_i^t - {\bm w}_i^\star \big\|^2 = {\cal O}( \sigma^t ),~~ (1/N) \sum_{i=1}^N \big\| \bm{\theta}_i^t - \overline{\bm{\theta}}(t) \big\| = {\cal O}( \sigma^t ) \;.

If N,M \gg 1 and the graph is geometric, a sufficient condition for convergence is to set \gamma = {\cal O}(1/ \max\{N^{2},M^2\}) and the resultant rate is \sigma = 1 - {\cal O}( 1/ \max\{M N^{2}, M^3\}).

Uses Notation · Used by Proof

Proof Sketch

Proof Sketch (Sketch)

QFJG

Our proof is divided into three steps. The first step studies the progress made by the algorithm in one iteration, taking into account the non-idealities due to imperfect tracking of the gradient over space and time. This leads to the characterization of a Lyapunov vector. The second step analyzes the coupled system of one iteration progress made by the Lyapunov vector. An interesting feature of it is that it consists of a series of independently delayed terms in the Lyapunov vector. The latter is resulted from the incremental update schemes employed in the method. Here, we study a sufficient condition for the coupled and delayed system to converge linearly. The last step is to derive condition on the step size \gamma_1 where the sufficient convergence condition is satisfied.

Specifically, we study the progress of the Lyapunov functions:

\begin{array}{c} \| \widehat{\underline{\bm v}}(t) \|^2 \mathrel{\mathop:}= \Theta \Big( \big\| \overline{\bm{\theta}}(t) - \bm{\theta}^\star \big\|^2 + (1/\beta N) \sum_{i=1}^N \big\| {\bm w}_i^t - {\bm w}_i^\star \big\|^2 \Big),~~ {\cal E}_c(t) \mathrel{\mathop:}= \frac{1}{N} \sqrt{\sum_{i=1}^N \| \bm{\theta}_i^t - \overline{\bm{\theta}}(t) \|^2}, \vspace{.1cm} \\ \textstyle {\cal E}_g(t) \mathrel{\mathop:}= \frac{1}{N} \sqrt{ \sum_{i=1}^N \big\| {\bm s}_i^t - \frac{1}{NM} \sum_{j=1}^N \sum_{p=1}^M {\nabla}_{\bm{\theta}} J_{j,p} ( \bm{\theta}_j^{\tau_p^t}, {\bm w}_j^{\tau_p^t} ) \big\|^2} \;. \end{array}

That is, \widehat{\underline{\bm v}}(t) is a vector whose squared norm is equivalent to a weighted distance to the optimal primal-dual solution, {\cal E}_c(t) and {\cal E}_g(t) are respectively the consensus errors of the primal parameter and of the primal aggregated gradient. These functions form a non-negative vector which evolves as:

\left( \begin{array}{c} \| \widehat{\underline{\bm v}}(t+1) \| \\[.1cm] {\cal E}_c(t+1) \\[.1cm] {\cal E}_g(t+1) \end{array} \right) \leq {\bm Q} (\gamma_1) \left( \begin{array}{c} \max_{ (t-2M)_+ \leq q \leq t } \| \widehat{\underline{\bm v}}(q) \| \\[.1cm] \max_{ (t-2M)_+ \leq q \leq t } {\cal E}_c(q) \\[.1cm] \max_{ (t-2M)_+ \leq q \leq t } {\cal E}_g(q) \end{array} \right) \;,

where the matrix {\bm Q} ( \gamma_1 ) \in \mathbb{R}^{3 \times 3} is defined by (exact form given in the supplementary material)

{\bm Q} ( \gamma_1 ) = \left( \begin{array}{ccc} 1 - \gamma_1 a_0 + \gamma_1^2 a_1 & \gamma_1 a_2 & 0 \\ 0 & \lambda & \gamma_1 \\ \gamma_1 a_3 & a_4 + \gamma_1 a_5 & \lambda + \gamma_1 a_6 \end{array} \right) \;.

In the above, \lambda \mathrel{\mathop:}= \lambda_{\sf max} ( {\bm W} - (1/N) {\bf 1}{\bf 1}^\top ) < 1, and a_0, ..., a_6 are some non-negative constants that depends on the problem parameters N, M, the spectral properties of {\bm A}, {\bm C}, etc., with a_0 being positive. If we focus only on the first row of the inequality system, we obtain

\begin{aligned} \| \widehat{\underline{\bm v}}(t + 1) \| \leq \big( 1 - \gamma_1 a_0 + \gamma_1^2 a_1 ) \max_{ (t-2M)_+ \leq q \leq t } \| \widehat{\underline{\bm v}}(q) \| + \gamma_1 a_2 \max_{ (t-2M)_+ \leq q \leq t } {\cal E}_c(q) \;. \end{aligned}

In fact, when the contribution from {\cal E}_c(q) can be ignored, then applying [feyzmahdavian2014delayed, Lemma 3] shows that \| \widehat{\underline{\bm v}}(t + 1) \| converges linearly if - \gamma_1 a_0 + \gamma_1^2 a_1 < 0, which is possible as a_0 > 0. Therefore, if {\cal E}_c(t) also converges linearly, then it is anticipated that {\cal E}_g(t) would do so as well.

In other words, the linear convergence of \| \widehat{\underline{\bm v}}(t) \|, {\cal E}_c(t) and {\cal E}_g(t) are all coupled in the inequality system (?).

Formalizing the above observations, Lemma 1 in the supplementary material shows a sufficient condition on \gamma_1 for linear convergence. Specifically, if there exists \gamma_1>0 such that the spectral radius of {\bm Q}( \gamma_1 ) in (?) is strictly less than one, then each of the Lyapunov functions, \| \widehat{\underline{\bm v}}(t) \|, {\cal E}_c(t), {\cal E}_g(t), would enjoy linear convergence. Furthermore, Lemma 2 in the supplementary material gives an existence proof for such an \gamma_1 to exist. This concludes the proof.

Uses Lemma 1, Lemma 2

Remark

KTOQ

While delayed inequality system has been studied in [feyzmahdavian2014delayed, gurbuzbalaban2017convergence] for optimization algorithms, the coupled system in (?) is a non-trivial generalization of the above. Importantly, the challenge here is due to the asymmetry of the system matrix {\bm Q} and the maximum over the past sequences on the right hand side are taken independently. To the best of our knowledge, our result is the first to characterize the (linear) convergence of such coupled and delayed system of inequalities.

Theorem

4869

If N,M \gg 1 and the graph is geometric with \lambda = 1 - c/N for c>0, a sufficient condition for convergence is to set \gamma = {\cal O}(1/ \max\{N^2,M^2\}) and the resultant rate is \sigma = 1 - {\cal O}( 1/ \max\{M N^2, M^3\}).

Uses Notation

Notation

PM8Y

We first define a set of notations pertaining to the proof. For any \beta > 0, observe that the primal-dual optimal solution, ( \bm{\theta}^\star, \{ {\bm w}_i ^{\star}\}_{i=1}^N ), to the optimization problem (?) can be written as

\underbrace{\left( \begin{array}{cccc} \rho {\bm I} & \sqrt{\frac{\beta}{N}} \hat{\bm A}^\top & \cdots & \sqrt{\frac{\beta}{N}} \hat{\bm A}^\top \\ -\sqrt{\frac{\beta}{N}} \hat{\bm A} & \beta \hat{\bm C} & \cdots & \cdots \\ \vdots & {\bm 0} & \ddots & {\bm 0} \\ -\sqrt{\frac{\beta}{N}} \hat{\bm A} & \cdots & \cdots & \beta \hat{\bm C} \end{array} \right)}_{\mathrel{\mathop:}= {\bm G}} \left( \begin{array}{c} \bm{\theta}^\star \\ \frac{1}{\sqrt{\beta N}}{\bm w}_1^\star \\ \vdots \\ \frac{1}{\sqrt{\beta N}}{\bm w}_N^\star \end{array} \right) = \left( \begin{array}{c} {\bm 0} \\ -\sqrt{\frac{\beta}{N}}{\bm b}_1 \\ \vdots \\ -\sqrt{\frac{\beta}{N}}{\bm b}_N \end{array} \right) \;,

where we denote the matrix on the left hand side as {\bm G}. This equation can be obtained by checking the first-order optimality condition. In addition, for any p \in \{ 1, \ldots, M \}, we define the {\bm G}_p as

{\bm G}_p \mathrel{\mathop:}= \left( \begin{array}{cccc} \rho {\bm I} & \sqrt{\frac{\beta}{N}} {\bm A}_p^\top & \cdots & \sqrt{\frac{\beta}{N}} {\bm A}^\top \\ -\sqrt{\frac{\beta}{N}}{\bm A}_p & \beta {\bm C}_p & \cdots & \cdots \\ \vdots & {\bm 0} & \ddots & {\bm 0} \\ -\sqrt{\frac{\beta}{N}}{\bm A}_p & \cdots & \cdots & \beta {\bm C}_p \end{array} \right) \;.

By definition, {\bm G} is the sample average of \{ {\bm G}_p \}_{p=1}^M. Define \bar{\bm{\theta}}(t) \mathrel{\mathop:}= (1/N) \sum_{i=1}^N \bm{\theta}_i^t as the average of the local parameters at iteration t. Furthermore, we define

\begin{aligned} {\bm h}_{\bm{\theta}}(t) \mathrel{\mathop:}= \rho \bar{\bm{\theta}}(t) + \frac{1}{N} \sum_{i=1}^N \hat{\bm A}^\top {\bm w}_i^t, & \qquad {\bm g}_{\bm{\theta}}(t) \mathrel{\mathop:}= \frac{1}{NM} \sum_{i=1}^N \sum_{p=1}^M \big( \rho \bm{\theta}_i^{\tau_p^t} + {\bm A}_p^\top {\bm w}_i^{\tau_p^t} \big) \;, \\ {\bm h}_{ {\bm w}_i } (t) \mathrel{\mathop:}= \hat{\bm A} \bar{\bm{\theta}}(t) - \hat{\bm C} {\bm w}_i^t - \hat{\bm b}_i, & \qquad {\bm g}_{ {\bm w}_i } (t) \mathrel{\mathop:}= \frac{1}{M} \sum_{p=1}^M \big( {\bm A}_p \bm{\theta}_i^{\tau_p^t} - {\bm C}_p {\bm w}_i^{\tau_p^t} - {\bm b}_{p,i} \big) \;, \end{aligned}

where {\bm h}_{\bm{\theta}}(t) and {\bm h}_{\bm w}(t) : = [{\bm h}_{ {\bm w}_1} (t), \cdots, {\bm h}_{ {\bm w}_N } (t)] represent the gradients evaluated by a centralized and batch algorithm. Note that {\bm g}_{\bm{\theta}}(t) defined in (Notation) coincides with that in (1). Using Lemma 1, it can be checked that \bar{\bm{\theta}}(t+1) = \bar{\bm{\theta}}(t) - \gamma_1 {\bm g}_{\bm{\theta}}(t) and {\bm w}_i^{t+1} = {\bm w}_i^t - \gamma_2 {\bm g}_{ {\bm w}_i } (t) for all t \geq 1. That is, \bar{\bm{\theta}}(t+1) and {\bm w}_i^{t+1} can be viewed as primal-dual updates using {\bm g}_{\bm{\theta}}(t) and {\bm g}_{ {\bm w}_i } (t), which are decentralized counterparts of gradients {\bm h}_{\bm{\theta}}(t) and {\bm h}_{ {\bm w}_i} (t) defined in (Notation) (Notation).

Uses Lemma 1, Notation

Lemma

Lemma 2

QMOM

Consider a sequence of non-negative vectors \{ {\bm e}(t) \}_{t \geq 1} \subseteq \mathbb{R}^n whose evolution is characterized by {\bm e}(t+1) \leq {\bm Q}\!~ {\bm e}( [ (t-M+1)_+ , t ] ) for all t \geq 1 and some fixed integer M>0, where {\bm Q} \in \mathbb{R}^{n \times n} is a matrix whose entries are nonnegative, and we define

{\bm e}( {\cal S} ) \mathrel{\mathop:}= \left( \begin{array}{c} \max_{ q \in {\cal S} } ~e_1 (q) \\ \vdots \\ \max_{ q \in {\cal S} } ~e_n(q) \end{array} \right) \in \mathbb{R}^n \qquad \text{for any subset}~{\cal S} \subseteq \mathbb{N}\; .

Moreover,

if {\bm Q} irreducible in the sense that there exists an integer m such that the entries of {\bm Q}^m are all positive, and the spectral radius of {\bm Q}, denoted by \rho( {\bm Q} ), is strictly less than one, then for any t\geq 1, we have

{\bm e}(t) \leq \rho({\bm Q})^{\lceil \frac{t-1}{ M} \rceil} C_1 {\bm u}_1 \;,

where {\bm u}_1 \in \mathbb{R}_{++}^n is the top right eigenvector of {\bm Q} and C_1 is a constant that depends on the initialization.

Uses Notation · Used by Proof Sketch

Proof of Lemma 2

We shall prove the lemma using induction. By the Perron-Frobenius theorem, the eigenvector {\bm u}_1 associated with \rho({\bm Q}) is unique and is an all-positive vector. Therefore, there exists C_1 such that

{\bm e}(1) \leq C_1\!~ {\bm u}_1 \;.

Let us first consider the base case with t=2,...,M+1, i.e., \lceil (t-1) / { M} \rceil = 1. When t=2, by (Proof) we have,

{\bm e}(2) \leq {\bm Q} {\bm e}(1) \leq C_1\!~ {\bm Q}\!~ {\bm u}_1 = \rho( {\bm Q})\!~ C_1\!~ {\bm u}_1 \;,

which is valid as {\bm Q}, {\bm e}(1), {\bm u}_1 are all non-negative. Furthermore, we observe that {\bm e}(2) \leq C_1 {\bm u}_1. Next when t = 3, we have

{\bm e}(3) \leq {\bm Q} {\bm e}([1,2]) \overset{(a)}{\leq} C_1\!~ {\bm Q}\!~ {\bm u}_1 = \rho( {\bm Q})\!~ C_1\!~ {\bm u}_1 \;,

where (a) is due to the non-negativity of vectors/matrix and the fact {\bm e}(1), {\bm e}(2) \leq C_1 {\bm u}_1 as shown in (?). Telescoping using similar steps, one can show {\bm e}(t) \leq \rho( {\bm Q} )\!~ C_1\!~ {\bm u}_1 for any t=2,...,M+1.

For the induction step, let us assume that (?) holds true for any t up to t= pM + 1. That is, we assume that the result holds for all t such that \lceil (t-1) / { M} \rceil \leq p. We shall show that it also holds for any t = pM+2, ..., (p+1)M + 1, i.e., \lceil (t-1) / { M} \rceil = p+ 1. Observe that

{\bm e}(pM+2) \leq {\bm Q} \!~ {\bm e}( [ (p-1)M + 2, pM+1 ] ) \leq C_1 \!~ \rho( {\bm Q})^p {\bm Q} {\bm u}_1 = \rho( {\bm Q})^{p+1}\!~ C_1 \!~ {\bm u}_1 \;,

where we have used the induction hypothesis. It is clear that (?) is equivalent to (?) with t = pM+2.

Similar upper bound can be obtained for {\bm e}(pM+3) as well. Repeating the same steps, we show that (?) is true for any t = pM+2, ..., (p+1)M + 1. Therefore, we conclude the proof of this lemma.

Q.E.D.

The following Lemma shows that {\bm Q} defined in (?) satisfies the conditions required in the previous lemma. Combining these two lemmas yields the final step of the proof of Theorem 1.

Lemma.

Consider the matrix {\bm Q} defined in (?), it can be shown that (a) {\bm Q} is an irreducible matrix in \mathbb{R}^{3 \times 3}; (b) there exists a sufficiently small \gamma such that \rho( {\bm Q}) < 1; and (c) as N,M \gg 1 and the graph is geometric, we can set \gamma = {\cal O}( 1 / \max\{N^2,M^2\}) and \rho( {\bm Q}) \leq 1 - {\cal O}( 1 / \max\{N^2,M^2\}).

Uses Theorem 1

Proof

U7SY

Our proof is divided into three parts. The first part shows the straightforward irreducibility of {\bm Q}; the second part gives an upper bound to the spectral radius of {\bm Q}; and the last part derives an asymptotic bound on \rho ( {\bm Q}) when N,M \gg 1.

Source and licence

The text on this page is from arXiv:1806.00877v4 by Hoi-To Wai, Zhuoran Yang, Zhaoran Wang and Mingyi Hong, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/1806/1806.00877.jsonl.