Convex and nonconvex nonparametric frontier-based classification methods for anomaly detection

Efective methods for determining the boundary of the normal class are very useful for detecting anomalies in commercial or security applications—a problem known as anomaly detection. This contribution proposes a nonparametric frontier-based classifcation (NPFC) method for anomaly detection. By relaxing the commonly used convexity assumption in the literature, a nonconvex-NPFC method is constructed and the nonconvex nonparametric frontier turns out to provide a more conservative boundary enveloping the normal class. By refecting on the monotonic relation between the characteristic variables and the membership, the proposed NPFC method is in a more general form since both input-like and outputlike characteristic variables are incorporated. In addition, by allowing some of the training observations to be misclassifed, the convex- and nonconvex-NPFC methods are extended from a hard nonparametric frontier to a soft one, which also provides a more conservative boundary enclosing the normal class. Both simulation studies and a real-life data set are used to evaluate and compare the proposed NPFC methods to some well-established methods in the literature. The results show that the proposed NPFC methods have competitive classifcation performance and have consistent advantages in detecting abnormal samples, especially the nonconvex-NPFC methods.