Formalizing the Use of the Activation Function in Neural Inference
Dalton A. R. Sakthivadivel
VERSES Research Lab, Los Angeles, CA, 90016, USA
Department of Mathematics
Department of Physics and Astronomy
Department of Biomedical Engineering
Stony Brook University, Stony Brook, NY, 11794, USA
dalton.sakthivadivel@stonybrook.edu
Abstract
We investigate how the activation function can be used to describe neural firing in an abstract way, and in turn, why it works well in artificial neural networks. We discuss how a spike in a biological neuron belongs to a particular universality class of phase transitions in statistical physics. We then show that the artificial neuron is, mathematically, a mean-field model of biological neural membrane dynamics, which arises from modeling spiking as a phase transition. This allows us to treat selective neural firing in an abstract way and formalize the role of the activation function in perceptron learning. The resultant statistical physical model allows us to recover the expressions for some known activation functions as various special cases. Along with deriving this model and specifying the analogous neural case, we analyze the phase transition to understand the physics of neural network learning. Together, it is shown that there is not only a biological meaning but a physical justification for the emergence and performance of typical activation functions; implications for neural learning and inference are also discussed.
Keywords: neural networks; Ising model; phase transitions; perceptrons
Cite this publication as:
D. A. R. Sakthivadivel, “Formalizing the Use of the Activation Function in Neural Inference,” Complex Systems, 31(4), 2022 pp. 433–449.
https://doi.org/10.25088/ComplexSystems.31.4.433
