Certified machine learning: A posteriori error estimation for physics-informed neural networks

Birgit Hillebrecht and Benjamin Unger

arXiv: 2203.17055v3 · PDF · 2 August 2022
Categories: cs.LG, cs.NA, math.NA
Journal: Proceedings of 2022 IEEE International Joint Conference on Neural Networks (IJCNN) (2022), 1-8
DOI: 10.1109/IJCNN55064.2022.9892569
Licence: CC BY 4.0 The site shows the text.

Abstract

Physics-informed neural networks (PINNs) are one popular approach to incorporate a priori knowledge about physical systems into the learning framework. PINNs are known to be robust for smaller training sets, derive better generalization problems, and are faster to train. In this paper, we show that using PINNs in comparison with purely data-driven neural networks is not only favorable for training performance but allows us to extract significant information on the quality of the approximated solution. Assuming that the underlying differential equation for the PINN training is an ordinary differential equation, we derive a rigorous upper limit on the PINN prediction error. This bound is applicable even for input data not included in the training phase and without any prior knowledge about the true solution. Therefore, our a posteriori error estimation is an essential step to certify the PINN. We apply our error estimator exemplarily to two academic toy problems, whereof one falls in the category of model-predictive control and thereby shows the practical use of the derived results.

Items 15

Notation

9V7L

The real numbers and the set of positive real numbers are denoted with \ensuremath\mathbb{R} and \ensuremath\mathbb{R}_+ \vcentcolon= \{x\in \ensuremath\mathbb{R} \mid x> 0\}, respectively. For a complex number \lambda\in\ensuremath\mathbb{C} we use the notation \Re(\lambda) to denote the real part of \lambda. The set of n\times m matrices with real coefficients is denoted with \ensuremath\mathbb{R}^{n\times m}. For the derivative w.r.t. to time we use interchangeably either the classical notation \frac{\mathrm{d}}{\mathrm{d}t} or the dot-notation prevalent in the dynamical systems literature, i.e., \dot{g}(t) \vcentcolon= \frac{\mathrm{d}}{\mathrm{d}t} g(t). The derivative with respect to the variable x = (x_1,\ldots,x_n)\in\ensuremath\mathbb{R}^n is denoted by \nabla_x f (x) = \sum_{i=1}^n \tfrac{\partial}{\partial x_i} f(x) \cdot \mathbf{i} for x \in \ensuremath\mathbb{R}^n with \mathbf{i}\in \ensuremath\mathbb{R}^n being the unit vector in the i-th direction and \tfrac{\partial}{\partial x_i} f the partial derivative of f with respect to the i-th component of x. We use the symbol \|\cdot\| to denote the Euclidean 2-norm. We emphasize that in the theoretical part, any other norm can be used. If the symbol \|\cdot\| is used on a set it describes the number of elements in this set.

Assumption

Assumption II.1

NQK0

The function f in (?) is continuous. Besides, it is Lipschitz-continuous with respect to the second argument, i.e., there exists a constant L>0 such that

\|f(t,x)-f(t,y)\|\leq L\|x-y\|

for all t\in\mathbb{T} and all x,y\in\ensuremath\mathbb{R}^n.

Used by Theorem III.1, Proof

Problem

Problem II.2

4WWP

Rigorously quantify the ML prediction error (?) for any t\in\mathbb{T} at low computational cost without computing the true solution.

Theorem

Theorem III.1

342A

Suppose f in (?) satisfies Assumption II.1 and the ML candidate function \hat{\varphi} is sufficiently smooth. For any continuous function \delta\colon\mathbb{T}\to\ensuremath\mathbb{R}_+ with

\|\mathcal{R}_{\hat{\varphi}}(t)\| \leq \delta(t)

define

I(t,\delta)\vcentcolon=\int_{0}^t \mathrm{e}^{L(t-s)}\delta(s)\,\mathrm{d}s,

where L is the Lipschitz constant from Assumption II.1. Then the ML prediction error (?) satisfies

|| e(t) || \leq \mathrm{e}^{L t} \|x_0-\hat{x}(0)\| + I(t,\delta).

Uses Assumption II.1 · Used by Remark III.2

Proof of Theorem III.1

Let \hat{x}_0 \vcentcolon= \hat{x}(0). Assumption II.1 together with the smoothness of \hat{\varphi} implicates that \mathcal{R}_{\hat{\varphi}} is continuous. Then, Assumption II.1 implies that \hat{x} is the unique solution of the perturbed initial value problem

\begin{aligned} \dot{\hat{x}}(t) &= f(t,\hat{x}(t)) + \mathcal{R}_{\hat{\varphi}}(t), & t\in\mathbb{T}, \\ \hat{x}(0) &= \hat{x}_0. \end{aligned}

The result is thus an immediate consequence of [HaiNW08, Ch. I, Variant of Thm. 10.2].

Uses Assumption II.1

Remark

Remark III.2

TDIA

Theorem III.1 can be easily generalized for time dependent Lipschitz parameters

|| f(t, x) - f(t, y) || \le \ell(t) || x-y||

by replacing the exponent Lt by L(t)=\int_0^t \ell(s)\,\mathrm{d}s in (III.1).

Uses Theorem III.1

Theorem

Theorem III.3

657Q

Suppose that the right-hand side of (?) is linear, i.e., given by (?), and \hat{\varphi} is sufficiently smooth. Then there exist constants \alpha \in\ensuremath\mathbb{R}, \beta\in\ensuremath\mathbb{R}_+ such that for any continuous function \delta\colon\mathbb{T}\to\ensuremath\mathbb{R} with

\|\mathcal{R}_{\hat{\varphi}}(t)\| \leq \delta(t),

the ML prediction error satisfies

\resizebox{.98\linewidth}{!}{$\begin{aligned} || e(t) || \leq \beta \left(\mathrm{e}^{\alpha t}\|x_0 -\hat{x}(0)\| + \int_{0}^t \mathrm{e}^{\alpha(t-s)}\delta(s)\,\mathrm{d}s\right). \end{aligned}$}

Proof of Theorem III.3

redCorrected Subtracting (Proof) from (?) yields the error dynamics

\begin{aligned} \dot{e}(t) &= Ae(t) + \mathcal{R}_{\hat{\varphi}}(t), & t\in\mathbb{T}, \\ e(0) &= x_0 - \hat{x}_0. \end{aligned}

Its unique solution is given by

e(t) = \mathrm{e}^{At}(x_0-\hat{x}_0) + \int_0^t \exp(A(t-s))\mathcal{R}_{\hat{\varphi}}(s)\,\mathrm{d}s.

Since for any matrix A there exist constants \alpha \in \ensuremath\mathbb{R}, \beta\in\ensuremath\mathbb{R}_+ such that \|\exp(At)\|\leq \beta \mathrm{e}^{\alpha t}, we obtain

\begin{aligned} \|e(t) \| &\leq \| \mathrm{e}^{At} (x_0-\hat{x}_0) \| + \int_0^t \| \exp(A(t-s))\mathcal{R}_{\hat{\varphi}}(s) \|\,\mathrm{d}s \\ &\le \beta\mathrm{e}^{\alpha t} \| x_0 -\hat{x}_0\| + \beta\int_0^t \mathrm{e}^{\alpha(t-s)} \delta(s) \,\mathrm{d}s \end{aligned}

which concludes the proof.

Uses Proof

Remark

Remark III.4

I44C

Theorem III.3 details a first step towards a generalization of the error estimator to PDEs. Assume that (?) constitutes a linear abstract ODE defined on an appropriate Banach space and that A in (?) is the generator of a strongly continuous semigroup (cf. [Paz83]). Then, assuming sufficiently smooth initial data, the error estimation (III.3) is still valid. Nevertheless, an additional approximation step is required to evaluate the norm on the underlying infinite-dimensional space. This is subject to further research.

Lemma

Lemma III.5

D7XL

Assume that the right-hand side f in (?) satisfies Assumption II.1 and that the ML candidate function \hat{\varphi} is sufficiently smooth. Let \delta \in \mathcal{C}^3(\mathbb{T}, \ensuremath\mathbb{R}) with

\|\mathcal{R}_{\hat{\varphi}}(t) \| \le \delta(t)

and \| \frac{\mathrm{d}^2}{\mathrm{d}s^2} \left( \mathrm{e}^{-Ls} \delta(s) \right)\| \le K for s \in \mathbb{T}. Then

\| \hat{x}(t) - x(t) \| \le \mathrm{e}^{Lt} \|x_0 - \hat{x}(0)\| + \hat{I}_n(t, \delta) + E_\mathrm{Int} \, \text{,}

with \hat{I}_n(t,\delta) is the composite trapezoidal rule approximation of I(t,\delta) using n subintervals. The error from the numerical integration is encapsulated in

E_\mathrm{Int} = \mathrm{e}^{Lt}\frac{Kt^3}{12 n^2}

Proof of Lemma III.5

Follows directly from the application of composite trapezoidal rule for the integral J(t, \delta) = I(t, \delta) \cdot e^{-Lt} and applying known error bounds [Gau97] to \hat{J}_n(t,\delta) and rescaling the result by multiplication with e^{Lt}.

Remark

Remark IV.1

7PZ3

If the underlying dynamic is linear and time-invariant, i.e., as in the setting of Theorem III.3, then instead of learning the error indicator, we could also use a PINN approach to directly learn the error dynamics (Proof). This should be closer to the actual prediction error since we do not introduce any additional approximation steps. Nevertheless, a thorough investigation is subject to further research.

Uses Proof

Remark

Remark V.1

L86F

When choosing \eta(t) to enhance the importance of small or large times during training, better results might be achieved w.r.t. two quality measures: firstly, the overall approximation error of the PINN might be smaller than without weighting, and secondly, the a posteriori error estimator might be closer to the actual error. These preliminary observations hint at the need for a thorough investigation of a weighting function \eta(t) in the context of certified machine learning and the introduced a posteriori error estimator.

Remark

Remark V.2

MZGR

We emphasize that we could also investigate the toy example with an explicit dependency on the initial value. This would introduce an additional input neuron and thus complicate the approximation task slightly. Nevertheless, the computation of the error bound and hence the certification of the approximation can be carried out as presented above. A more realistic example with a varying initial value is discussed next.

Remark

Remark V.3

2H90

In the previous examples, results were presented which use the hyperbolic tangent as the activation function. However, the choice of the activation function does not affect the validity or the applicability of the estimator. Tests with other activation functions such as gaussian error linear unit (gelu), softmax, or sigmoid-weighted linear units (silu) showed similar results. The same holds for the choice of the optimizer.

Source and licence

The text on this page is from arXiv:2203.17055v3 by Birgit Hillebrecht and Benjamin Unger, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2203/2203.17055.jsonl.