The Main Idea

Gradient based methods can be viewed as relaxation methods for the equation

$${\cal H} \alpha = g$$ (1)

where $g$ is the gradient and ${\cal H}$ is the Hessian of the functional considered. For example, a Jacobi relaxation for equation (1) has the form

$$\alpha \leftarrow \alpha + \delta (g - {\cal H} \alpha )$$ (2)

which is essentially the steepest descent method for minimizing the cost functional. The observation that the convergence rate for gradient descent methods is governed by $I - \delta {\cal H}$ suggests that effective Preconditioners can be constructed using the behavior of the symbol of the Hessian $\hat{\cal H}({\bf k})$, for large ${\bf k}$. The idea is simple. Assume that

$$\hat{\cal H} ({\bf k}) = O( \vert{\bf k}\vert^\gamma) \qquad \mbox{\rm for large $\vert{\bf k}\vert$}$$ (3)

and let ${\cal R}$ be an operator whose symbol satisfies

$$\hat{\cal R} ({\bf k}) \ = O( \frac{1}{\vert{\bf k}\vert^\gamma}) \qquad \mbox{\rm for large $\vert{\bf k}\vert$}.$$ (4)

The behavior of the preconditioned method

$$\alpha \leftarrow \alpha - \delta {\cal R} g$$ (5)

is determined by

$$I - \delta {\cal R} {\cal H}$$ (6)

whose symbol

$$1 - \delta \hat{\cal R} ({\bf k})\hat{\cal H} ({\bf k})$$ (7)

approaches a constant for large $\vert{\bf k}\vert$. A proper choice of $\delta$ leads to a convergence rate which is independent of the dimensionality of the design space. This is not the case if the symbol of the iteration operator has some dependence on ${\bf k}$.

It is desired not to change the behavior of the low frequencies by the use of the preconditioner, since the analysis we do for the Hessian does not hold in the limit $\vert {\bf k} \vert \rightarrow 0$. That is, we would like the symbol of the preconditioner to satisfy also,

$$\hat{\cal R} ( {\bf k} ) \rightarrow 1 \quad \mbox{ for } \vert {\bf k} \vert \rightarrow 0.$$ (8)