Nonlinear problems in analysis of Krylov subspace methods

Consider a system of linear algebraic equations $Ax=b$ where $A$ is an $n$ by $n$ real matrix and $b$ a real vector of length $n$. Unlike in the linear iterative methods based on the idea of splitting of $A$, the Krylov subspace methods, which are used in computational kernels of various optimization techniques, look for some optimal approximate solution $x^n$ in the subspaces ${\cal K}_n (A, b) = \mbox{span} \{ b, Ab, \dots, A^{n-1}b\}, n = 1, 2, \dots$ (here we assume, with no loss of generality, $x^0 = 0$). As a consequence, though the problem $Ax = b$ is linear, Krylov subspace methods are not. Their convergence behaviour cannot be viewed as an (unimportant) initial transient stage followed by the subsequent convergence stage. Apart from very simple, and from the point of view of Krylov subspace methods uninteresting cases, it cannot be meaningfully characterized by an asymptotic rate of convergence. In Krylov subspace methods such as the conjugate gradient method (CG) or the generalized minimal residual method (GMRES), the optimality at each step over Krylov subspaces of increasing dimensionality makes any linearized description inadequate. CG applied to $Ax = b$ with a symmetric positive definite $A$ can be viewed as a method for numerical minimization the quadratic functional $1/2 (Ax, x) - (b,x)$. In order to reveal its nonlinear character, we consider CG a matrix formulation of the Gauss-Christoffel quadrature, and show that it essentially solves the classical Stieltjes moment problem. Moreover, though the CG behaviour is fully determined by the spectral decomposition of the problem, the relationship between convergence and spectral information is nothing but simple. We will explain several phenomena where an intuitive commonly used argumentation can lead to wrong conclusions, which can be found in the literature. We also show that rounding error analysis of CG brings fundamental understanding of seemingly unrelated problems in convergence analysis and in theory of the Gauss-Christoffel quadrature. In remaining time we demonstrate that in the unsymmetric case the spectral information is not generally sufficient for description of behaviour of Krylov subspace methods. In particular, given an arbitrary prescribed convergence history of GMRES and an arbitrary prescribed spectrum of the system matrix, there is always a system $Ax=b$ such that GMRES follows the prescribed convergence while $A$ has the prescribed spectrum.