Model fitting by least squares

The magical property of the conjugate direction method

In the conjugate-direction method, not a line, but rather a plane, is searched. A plane is made from an arbitrary linear combination of two vectors. One vector is chosen to be the gradient vector, say $\bold g$. The other vector is chosen to be the previous descent step vector, say $\bold s = \bold x_j - \bold x_{j-1}$. Instead of $\alpha \, \bold g$, we need a linear combination, say $\alpha \bold g + \beta \bold s$. For minimizing quadratic functions the plane search requires only the solution of a two-by-two set of linear equations for $\alpha$ and $\beta$.

The conjugate-direction (CD) method described in this book has a magical property shared by the more famous conjugate-gradient method. This magical property is not proven in this book, but it may be found in many sources. Although both methods are iterative methods, both converge on the exact answer (assuming perfect numerical precision) in a fixed number of steps. That number is the number of components in model space $\bold x$.

Where we benefit from iterative methods is where convergence is more rapid than the theoretical requirement. Whether or not that happens, depends on the problem at hand. Reflection seismology has many problems so massive they are said to be solved simply by one application of the adjoint operator. The idea that such solutions might be improved by a small number of iterations is very appealing.

Model fitting by least squares