TLS-Aware Anomaly Detection for Encrypted IoT Traffic Using a β -Variational Autoencoder with ANOVA–Mutual Information Feature Selection

The rapid growth of the Internet of Things (IoT) has increased dependency on Transport Layer Security (TLS) for securing device communications, enhancing confidentiality while reducing the visibility required by traditional intrusion detection systems. As payload inspection becomes impractical in encrypted environments, anomaly detection must instead rely on flow-level statistics and TLS metadata. This is challenging because IoT traffic is heterogeneous, non-stationary, and distributionally inconsistent across datasets, while many existing studies rely on single-dataset evaluation and therefore provide limited evidence of real-world generalisation. We introduce a TLS-aware anomaly detection framework that combines a β -Variational Autoencoder ( β -VAE) with a hybrid ANOVA–Mutual Information (ANOVA–MI) feature-selection pipeline. The incremental contribution lies not in the individual use of these components, but in their integrated application to encrypted IoT anomaly detection under strict cross-dataset evaluation, where feature filtering, probabilistic latent regularisation, and threshold transferability are jointly examined without retraining or recalibration on target datasets. The framework models benign encrypted IoT traffic using probabilistic latent representations and identifies anomalies through reconstruction-error-based scoring. Network flows from the BoT-IoT, IoT-23, and ToN-IoT datasets were processed using Zeek and CICFlowMeter to construct a unified metadata feature space incorporating flow statistics and TLS attributes such as JA3 and JA3S fingerprints. The model was trained on benign BoT-IoT traffic and evaluated in both in-dataset and cross-dataset scenarios. The model achieves strong in-dataset performance on BoT-IoT (ROC-AUC ≈ 0.9996 ; F1 ≈ 0.9922 ) and retains robust anomaly-ranking and threshold-based detection capability under cross-dataset domain shift (IoT-23: ROC-AUC ≈ 0.9882 , F1 ≈ 0.9422 ; ToN-IoT: ROC-AUC ≈ 0.9465 , F1 ≈ 0.8732 ). A comparative evaluation against deterministic autoencoders and classical baselines further indicates that the proposed β -VAE achieves stronger cross-dataset anomaly-ranking performance than the compared methods. These findings support the suitability of probabilistic latent modelling for privacy-preserving anomaly detection in encrypted IoT environments.