A hybrid autoencoder-random forest framework for open-set recognition of sensor and communication faults in a variable refrigerant flow system with real-time deployment analysis

Most variable refrigerant flow (VRF) system-based fault detection and diagnosis (FDD) models are closed-set and are evaluated offline. Therefore, focusing on faults affecting sensor and communication components in a VRF system, this work explicitly addresses both open-set recognition and real-time deployment and proposes a hybrid autoencoder–random-forest (AE-RF) framework. The AE, trained on normal data only, provides a reconstruction-error–based novelty score, while a calibrated RF, trained only on known classes, outputs fault posteriors. A decision-level fusion strategy combines an AE error threshold, a confidence gate on the maximum class probability and a top-two class probability margin to decide whether a sample is assigned to a known fault class or rejected as Unknown. The framework is benchmarked against a cascaded convolutional neural network-one class support vector machine (CNN–OCSVM) baseline model. During the independent offline test, the AE–RF framework achieves a normalized accuracy (NA) of 99.27 % and a Youden's index (J) of 88.33 %, outperforming CNN–OCSVM by 5.8% in NA and 211.8% in J in relative terms. In real-time deployment, the frozen AE–RF model maintains a high NA of 98.96 % and a J of 83.33 %, accurately tracking known faults while safely rejecting unfamiliar conditions. Ablation study shows that removing the margin rule or confidence gate markedly reduces both NA and J, and removing both reduces the model to a crude novelty detector, underscoring the importance of all fusion components for reliable open-set FDD in VRF systems.