The solution to a stochastic optimal control problem can be determined by computing the value function from a discretization of the associated Hamilton-Jacobi-Bellman equation. Alternatively, the problem can be reformulated in terms of a pair of forward-backward SDEs, which makes Monte-Carlo techniques applicable. More recently, the problem has also been viewed from the perspective of forward and reverse time SDEs and their associated Fokker-Planck equations. This approach is closely related to techniques used in diffusion-based generative models. Forward and reverse time formulations express the value function as the ratio of two probability density functions; one stemming from a forward McKean-Vlasov SDE and another one from a reverse McKean-Vlasov SDE. In this paper, we extend this approach to a more general class of stochastic optimal control problems and combine it with ensemble Kalman filter type and diffusion map approximation techniques in order to obtain efficient and robust particle-based algorithms.
The transformed HJB equation (?) simplifies to
for the special case G = \sigma and R = I and (?) becomes linear in v_t^\ast. This well-known fact has been exploited in the numerical work of [MO22]. Furthermore, the transformed HJB equation arising from the further simplification c(x)\equiv 0 leads to the standard backward Kolmogorov equation [Pavliotis2016].
Given the forward evolution equation (?) and the HJB equation (?), the probability density defined by (?) satisfies the reverse time evolution equation
with terminal condition \tilde \pi_T = Z_T^{-1}\exp(-f) \bar \pi_T, where \zeta_t is an appropriate normalization constant.
Since (?) and
and assuming that (?) holds, it follows that
Hence it holds indeed that
for the partial time derivatives given by (3.1), (?), and (?), respectively. Furthermore, \tilde \pi_T = v_T^\ast \bar \pi_T/Z_T at final time and, hence, (?) holds for all times t\in (0,T].
While [MRO20] and [MO22] form the basis for our work, we mention the alternative approach put forward in [JTMM22], where v_t^\ast is viewed directly as an unnormalized probability density. This approach leads to the interpretation of the HJB equation (?) in terms of a nonlinear Fokker–Planck equation which in turn can be approximated using interacting particles and EnKF-type approximations as in our work. Only deterministic control problems are considered in [JTMM22]. Furthermore, it is not obvious whether the value function v_t^\ast is always normalizable with respect to the Lebesque measure on \mathbb{R}^{d_x}.
The two diffusion processes defined by (?) satisfy \bar X_t \sim \bar \pi_t and \tilde X_t \sim \tilde \pi_t, respectively. Given \tilde \pi_t and \bar \pi_t, the optimal control law u_t^\ast(x) is provided by
The lemma follows immediately from writing down the associated (nonlinear) Fokker–Planck equations for the two McKean–Vlasov SDEs (?). The stated formula for the optimal control follows from (?).
The text on this page is from arXiv:2311.06906v4 by Sebastian Reich, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2311/2311.06906.jsonl.