Center for Advanced Intelligence Project

Speaker: Prof. Michael I. Jordan (University of California, Berkeley, USA)
https://people.eecs.berkeley.edu/~jordan/

Title:
Variational, Hamiltonian and Symplectic Perspectives on Acceleration

Abstract:
Accelerated gradient methods play a central role in optimization, achieving
optimal rates in many settings. While many generalizations and extensions of
Nesterov’s original acceleration method have been proposed, it is not yet
clear what is the natural scope of the acceleration concept. We study
accelerated methods from a continuous-time perspective. We show that there
is a Lagrangian functional that we call the “Bregman Lagrangian” which generates
a large class of accelerated methods in continuous time, including (but not
limited to) accelerated gradient descent, its non-Euclidean extension, and
accelerated higher-order gradient methods. We show that the continuous-time
limit of all of these methods correspond to traveling the same curve in
spacetime at different speeds. We also describe a “Bregman Hamiltonian”
which generates the accelerated dynamics, we develop a symplectic integrator
for this Hamiltonian and we discuss relations between this symplectic integrator
and classical Nesterov acceleration. I also discuss some of the implications
of this geometric perspective for efficient saddle-point avoidance.