Objective acceleration for unconstrained optimization

Acceleration schemes can dramatically improve existing optimization
procedures. In most of the work on these schemes, such as nonlinear Generalized
Minimal Residual (N-GMRES), acceleration is based on minimizing the $\ell_2$
norm of some target on subspaces of $\mathbb{R}^n$. There are many numerical
examples that show how accelerating general purpose and domain-specific
optimizers with N-GMRES results in large improvements. We propose a natural
modification to N-GMRES, which significantly improves the performance in a
testing environment originally used to advocate N-GMRES. Our proposed approach,
which we refer to as O-ACCEL (Objective Acceleration), is novel in that it
minimizes an approximation to the \emph{objective function} on subspaces of
$\mathbb{R}^n$. We prove that O-ACCEL reduces to the Full Orthogonalization
Method for linear systems when the objective is quadratic, which differentiates
our proposed approach from existing acceleration methods. Comparisons with
L-BFGS and N-CG indicate the competitiveness of O-ACCEL. As it can be combined
with domain-specific optimizers, it may also be beneficial in areas where
L-BFGS or N-CG are not suitable.