Hierarchical Sparse Neural Networks for Structure-Aware Ransomware Detection Under Distribution Shift

Behavioral ransomware detection often achieves high accuracy under standard evaluation settings, but such results may not generalize under distribution shift or when previously unseen ransomware families are encountered. This study evaluates detection performance on the MLRan dataset, which contains 4880 samples from 64 ransomware families, using four evaluation protocols: stratified, temporal, family-disjoint, and open-set. The family-disjoint and open-set protocols were constructed at the family level to limit overlap between learned and held-out ransomware families. The study proposes the Hierarchical Sparse Neural Network (HSNN), a taxonomy-aligned model that uses group-level and branch-level gating to support structured interpretability and modality-level analysis. Compared with the FlatMLP baseline, HSNN achieved a slightly lower average macro-F1 score (0.9839 vs. 0.9860) but showed better calibration and lower model complexity. Specifically, HSNN reduced expected calibration error by 34.1% and parameter count by 42%. HSNN also showed slightly lower variability across random seeds and stable gate patterns. Under the open-set family protocol, HSNN achieved one of the strongest macro-F1 scores (0.9930 vs. 0.9913 for FlatMLP) using a maximum-softmax novelty baseline. Feature analysis indicates that string-based artifacts remain strong predictors, while the hierarchical structure distributes importance across multiple behavioral modalities. These results position HSNN as a competitive alternative to dense neural baselines when calibration, compactness, and structured interpretability are considered alongside macro-F1 performance.