Innovative Applications of Metaheuristics to Supervised Machine Learning

The optimization of hyperparameters and the learning process associated with supervised machine learning models, including artificial neural networks and deep learning architectures, are viewed as among the most formidable challenges in machine learning. Several prior investigations have applied gradient-based backpropagation approaches in the training of deep learning architectures. However, gradient-based methods exhibit notable disadvantages, such as the tendency to become trapped in local minima when dealing with multi-objective cost functions, extensive calculations of gradient information with numerous iterations, and requiring the cost functions to maintain continuity. Given that the training process for a diverse array of supervised machine learning techniques is identified as an NP-hard optimization problem, there has been a considerable surge in the adoption of heuristic or metaheuristic algorithms aimed at optimizing the structures and parameters of these approaches. In the context of optimization, metaheuristic optimizers are designed to identify the optimal estimations for different components of supervised machine learning (e.g., weights, hyperparameters, the number of layers or neurons, and the learning rate in case of deep learning models, or hyperparameters for Gaussian processes in Gaussian process regression (GPR), etc.). Many researchers are inclined to extend innovative hybrid algorithms by integrating metaheuristic algorithms to optimize the hyperparameters of supervised machine learning models. The evolution of hybrid metaheuristics aids in boosting algorithm performance and is proficient in addressing complex optimization problems. In general, the optimal functioning of the metaheuristics ought to attain a favorable compromise between the exploration and exploitation characteristics. This chapter has comprehensively reviewed two recent advancements in the application of metaheuristic optimizers in supervised machine learning approaches, which are distinctly diverse in their characteristics. To begin with, this chapter offers a detailed review of a modern hybrid intrusion detection technique that employs a black widow-optimized convolutional long short-term memory neural network. Following this, the authors provide an in-depth analysis of a recent investigation focused on the metaheuristic optimization of hyperparameters for the GPR model associated with FEREBUS, a GPR engine embedded within the extensive framework of FFLUX, which is recognized as an innovative machine-learned force field. The integration of metaheuristics with supervised machine learning models is projected to expedite the training process in the forthcoming years. However, the concept of evolutionary hybrid architecture remains underutilized in the existing literature, with relevant publications being relatively uncommon.