Definition 3.1
AAUAThe adapted vector field v^\ast(x)\in T_x\mathcal{M} of a vector v \in T_p\mathcal{M} is defined in a normal neighborhood x \in U_p by parallel transporting v along the unique geodesic between x and p.
Adaptive and Momentum Methods on Manifolds Through Trivializations
- arXiv
- 2010.04617v1 · PDF · 9 October 2020
- Categories
- cs.LG, cs.NA, math.NA, math.OC
- Licence
- CC BY 4.0 The site shows the text.
Abstract
Adaptive methods do not have a direct generalization to manifolds as the adaptive term is not invariant. Momentum methods on manifolds suffer from efficiency problems stemming from the curvature of the manifold. We introduce a framework to generalize adaptive and momentum methods to arbitrary manifolds by noting that for every differentiable manifold, there exists a radially convex open set that covers almost all the manifold. Being radially convex, this set is diffeomorphic to \mathbb{R}^n. This gives a natural generalization of any adaptive and momentum-based algorithm to a set that covers almost all the manifold in an arbitrary manifolds. We also show how to extend these methods to the context of gradient descent methods with a retraction. For its implementation, we bring an approximation to the exponential of matrices that needs just of 5 matrix multiplications, making it particularly efficient on GPUs. In practice, we see that this family of algorithms closes the numerical gap created by an incorrect use of momentum and adaptive methods on manifolds. At the same time, we see that the most efficient algorithm of this family is given by simply pulling back the problem to the tangent space at the initial point via the exponential map.
Items 8
Theorem 4.1
8A8NLet (\mathcal{M}, \textsl{g}) be a complete and connected Riemannian manifold. Let \overline{U}_p be the (closed) set of vectors v \in T_p\mathcal{M} for which \gamma(t) = \exp_p(tv), t \in [0,1] is length minimizing, and let U_p be its interior and C_p its boundary. Then, \exp_p is a diffeomorphism on U_p and \exp_p(C_p) has measure zero with respect to the Borel measure induced by \textsl{g}.
Used by Proof
Proof
L7P0See [petersen2016riemannian, Section 5.7.3] for the first part. The second follows from [itoh1998dimension].
Proposition 4.2
K5CWLet (\mathcal{M}, \textsl{g}) be a Riemannian manifold. Let f be any differentiable function f on \mathcal{M}, and let p_0,p_i \in \mathcal{M} be some points and v_{i,k} \in T_{p_i}\mathcal{M} with x_t \coloneqq \exp_{p_i}(v_{i,k}) such that p_0, p_i \in U_{x_t} with U_{x_t} as in Theorem 4.1. Then, the map
\dodeffunction\mathrm{d}\pa{\exp^{-1}_{p_0} \circ \exp_{p_i}}^\ast_{v_{i,k}} : T_{p_0}\mathcal{M} -> T_{p_i}\mathcal{M};\relax
were (-)^\ast is the adjoint, takes directions of steepest descent of f \circ \exp_{p_0} to directions of steepest descent of f \circ \exp_{p_i}. Furthermore, it is the only map between T_{p_0}\mathcal{M} and T_{p_i}\mathcal{M} with this property.
Proof of Proposition 4.2
The hypothesis on p_0, p_i and Theorem 4.1 assure that the operator \mathrm{d}\pa{\exp^{-1}_{p_0} \circ \exp_{p_i}}_{v_{i,k}} exists and is an isomorphism of linear spaces. By the chain rule, letting w_{i,k} \coloneqq \exp^{-1}_{p_0}(x_t) we have that
\mathrm{d} \pa{f \circ \exp_{p_i}}_{v_{i,k}} = \mathrm{d} \pa{f \circ \exp_{p_0}}_{w_{i,k}} \circ \mathrm{d} \pa{\exp_{p_0}^{-1} \circ \exp_{p_i}}_{v_{i,k}}
and taking adjoints
\begin{aligned}
\nabla\pa{f &\circ \exp_{p_i}}(v_{i,k}) =\\
&\mathrm{d}\pa{\exp_{p_0}^{-1} \circ \exp_{p_i}}^\ast_{v_{i,k}}\pa{\nabla \pa{f \circ \exp_{p_0}}\pa{w_{i,k}}}.\qedhere
\end{aligned}
Uses Theorem 4.1
Theorem 4.3
DIY9Let (\mathcal{M}, \textsl{g}) be a connected and complete Riemannian manifold with zeroth and first order curvature bounds \delta \leq \sec \leq \Delta and \norm{\nabla R} \leq \Lambda. Consider a connected submanifold \mathcal{X} with \operatorname{diam}(\mathcal{X}) = r where r \leq \frac{\sqrt{2}\pi}{\sqrt{\delta + \Delta}} if \Delta > - \delta, and let f be a function on \mathcal{X} bounded below by f^\ast \in \mathbb{R} such that \norm{\operatorname{Hess} f} \leq \alpha. Then, there exists a constant \widehat{\alpha}_r that does not depend on the dimension of \mathcal{M} such that \norm{\operatorname{Hess} \pa{f \circ \exp_p}\vert_{\exp_p^{-1}\pa{\mathcal{X}}}} \leq \widehat{\alpha}_r for any p \in \mathcal{X}. Therefore, running atrivK with gd and \eta = \frac{1}{\widehat{\alpha}_r}, if all the iterates stay in \mathcal{X}, the algorithm will find a point v_{i, k} \in T_{p_i}\mathcal{M} such that \norm{\nabla \pa{f \circ \exp_{p_i}}(v_{i,k})} < \varepsilon in at most \ceil[\Big]{\mbox{\@setsize\@setsize\@setsize\@setsize\@setsize\@setsize\@setsize\small{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt{10pt}\ixpt\@ixpt2_r^2}\pa{f(p_0) - f^\ast}} steps.
Proof of Theorem 4.3
The result follows from [lezcano2020curvature, Proposition 5.1] and the classical theory of non-convex optimization [nesterov2004introductory].
Remark (Copying and adding problems)
73IZWe do not include the adding problem or the copying problem [hochreiter1997long] which are often considered, as these problems are too simple for recurrent models [henaff2016recurrent]. All the considered optimization methods converge to a correct answer in the copying and adding problems.
Source and licence
The text on this page is from arXiv:2010.04617v1 by Mario Lezcano-Casado, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2010/2010.04617.jsonl.