Accelerated Backpropagation Learning: Two Optimization Methods
Roberto Battiti
Caltech Concurrent Computation Program,
206-49 California Institute of Technology, Pasadena, CA 91125, USA
Abstract
Two methods for increasing performance of the backpropagation learning algorithm are presented and their results are compared with those obtained by optimizing parameters in the standard method. The first method requires adaptation of a scalar learning rate in order to decrease the energy value along the gradient direction in a close-to-optimal way. The second is derived from the conjugate gradient method with inexact linear searches. The strict locality requirement is relaxed but parallelism of computation is maintained, allowing efficient use of concurrent computation. For medium-size problems, typical speedups of one order of magnitude are obtained.
