Performance-Driven Dimensionality Reduction: A Data-Centric Approach to Feature Engineering in Machine Learning

In a number of applications, data may be anonymized, obfuscated, and highly noisy. In such cases, it is difficult to use domain knowledge and low-dimensional visualizations to engineer the features for tasks such as machine learning. In this work, we explore a variety of dimensionality reduction (DR) techniques in the form of feature extraction and feature selection to decrease multicollinearity and improve the predictive power of our modeling tasks. These techniques include principal component analysis (PCA), locally linear embedding (LLE), Isomap, Kernel principal component analysis (KPCA), uniform manifold approximation and projection (UMAP), mean decrease accuracy, Shapley Values, and feature clustering. Due to the data-driven nature of our methodology, all forms of DR algorithm selection, hyperparameter tuning, and model tuning are done purely based on performance on our models, rather than a priori knowledge. This method will show which technique will increase the predictive power of our random forest model. Due to the generality of our method, this approach offers flexibility for regression or classification with any machine learning model and any unsupervised DR technique.