Using the Functional Behavior of Neurons for Genetic Recombination in Neural Nets Training
Nachum Shamir
Current address: Department of Electrical Engineering, Technion--Israel Institute of Technology, Technion City, Haifa 32000, Israel.
David Saad
Current address: Department of Physics, University of Edinburgh, J. C. Maxwell Building, Mayfield Road, Edinburgh EH9 3JZ, UK.
Emanuel Marom
Faculty of Engineering, Tel Aviv University,
Ramat Aviv 69978, Israel
Abstract
We propose a new hybrid genetic back propagation training algorithm based on a unique functional matching recombination method. The method is used to evolve populations of neural networks and provides versatility in network architecture and activation functions. Net reorganization and reconstruction is carried out prior to genetic recombination using a functional behavior correlation measure to compare the functional role of the various neurons. Comparison is done by correlating the internal representations generated for a given training set. Net structure is dynamically changed during the evolutionary process, expanded by reorganization and reconstruction and trimmed by pruning unnecessary neurons. The ability to change net structure throughout generations allows the net population to fit itself to the requirements of dynamic adaptation, performance, and size considerations in the selection process, thus generating smaller and more efficient nets that are likely to have higher generalization capabilities. A functional behavior correlation measure is used extensively to explore and compare nets and neurons, and its ability is demonstrated by investigating the results of genetic recombination. The vitality of nets organized via the functional behavior correlation measure prior to genetic recombination is demonstrated by statistical results of computer simulations. The performance of the proposed method and its generalization capabilities are demonstrated using Parity, Symmetry and handwritten digit recognition training tasks.
