Internal noise in analog neural networks helps with learning

Recently, the field of hardware neural networks has been actively developing, where neurons and their connections are not simulated on a computer but are implemented at the physical level, transforming a neural network into a tangible device. In terms of hardware neural networks, it is more important to consider not only the effect of noise on the input signal, but also the effect of internal noise coming from various network components. In this paper, we investigate how internal noise affects the final performance of feedforward neural networks (FNN) and echo state networks (ESN) during the training of neural networks. The types of noise considered in this paper were originally inspired by a real optical implementation of a neural network. However, these types were subsequently generalized to enhance the applicability of our findings on a broader scale. The noise types considered include additive and multiplicative noise, which depend on how noise influences each individual neuron, and common and uncommon noise, which pertains to the impact of noise on groups of neurons (such as the hidden layer of FNNs or the reservoir of ESNs). In this paper, we demonstrate that, in most cases, both deep and echo state networks benefit from internal noise during training, as it enhances their resilience to noise. Consequently, the testing performance at the same noise intensities is significantly higher for networks trained with noise than for those trained without it. Only multiplicative common noise during training has almost no impact on both deep and recurrent networks.