A Vision Based Deep Learning Framework for Malware Detection and Classification

Malware detection is a complex task for signature-based anti-virus software, especially for polymorphic malware and zero-day attacks. However, this project proposes a vision-based static malware detection and classification method that represents raw executable file bytes as fixed-size grayscale images called byte plots and attempts to classify malware families based on these images without executing them. In this project, for the proposed model, the best architecture is Convolutional Neural Networks (CNN) + Random Forest (CNN-RF). Initially, a CNN is trained to learn discriminative feature embeddings for byte plot images. Once this is done, the final softmax classifier is removed, and this CNN is used to generate a 256-dimensional vector for each input. Then, a class-balanced Random Forest is trained to predict the malware family and confidence scores. In this way, this proposed method is able to achieve better results for two different datasets, and the best results obtained are 98.07% for MalImg and 93.07% for MaleVis.