Optimizing Intrusion Detection Systems: A Machine Learning-Based Feature Selection Approach for Enhanced Cybersecurity

The rapid evolution of technologies such as artificial intelligence and big data has led to an increase in sophisticated cyberattacks, making network security an urgent concern. Intrusion Detection Systems (IDS) have been widely studied, with machine learning playing an important role in enhancing their effectiveness. However, developing a high-performing IDS remains challenging due to the complexity of high-dimensional data and limitations in detection accuracy. Effective feature selection is essential for improving IDS performance by reducing dimensionality, enhancing interpretability, and preventing overfitting. In this study, we propose a comprehensive data preprocessing pipeline to optimize feature selection and apply it to the UNSW-NB15 dataset, a benchmark for cybersecurity threat detection. We evaluate the impact of several feature selection techniques on four machine learning models: Logistic Regression, K-Nearest Neighbors (KNN), Decision Tree, and Random Forest. The selection methods include XGBoost-based selection with correlation analysis, Chi-square test, Recursive Feature Elimination (RFE), Fisher’s test and CatBoost-based selection. We analyze model robustness across various feature subset sizes, and evaluate performance using Accuracy, Precision, Recall, and F1-score. Our findings demonstrate that feature selection significantly impacts model performance, with Random Forest achieving the highest accuracy (94.96%) using only 25 features selected via XGBoost and correlation. These results highlight the importance of optimal feature selection for designing more effective IDS solutions.