Adaptive multilayer extreme learning machines

Extreme learning machines is a neural network type that has been utilized in tasks such as regression and classification, due to their efficient training process, which is based on pseudoinverse matrices and randomized weights, avoiding the computationally intensive backpropagation. In order to further improve their performance and reduce their complexity with respect to number of required hyperparameters, especially in the case of multiple layer architectures, a novel multilayer adaptive approach, based on residual networks, is proposed. This approach constructs the network iteratively with respect to error minimization and parsimony using a recursive pseudoinverse matrix framework. A new block approach, using mixed precision arithmetic and Graphics Processing Units (GPU) is proposed. The proposed technique is coupled with a new adaptive penalty criterion to ensure adequate numbers of neurons are included in each layer, while avoiding highly correlated basis. Adaptive regularization, along with scaling, is also incorporated to ensure Symmetric Positive Definiteness (SPD) of the Gram matrix. Several random number distributions for the proposed approach are examined and discussed. Handling of large datasets is discussed and a new batch variant is proposed. The proposed scheme is evaluated for regression and classification tasks in a multitude of datasets and is compared with other neural network architectures.