IDEAS home Printed from https://ideas.repec.org/a/oup/biomet/v97y2010i4p893-904.html
   My bibliography  Save this article

Consistent selection of the number of clusters via crossvalidation

Author

Listed:
  • Junhui Wang

Abstract

In cluster analysis, one of the major challenges is to estimate the number of clusters. Most existing approaches attempt to minimize some distance-based dissimilarity measure within clusters. This article proposes a novel selection criterion that is applicable to all kinds of clustering algorithms, including distance based or non-distance based algorithms. The key idea is to select the number of clusters that minimizes the algorithm's instability, which measures the robustness of any given clustering algorithm against the randomness in sampling.Anovel estimation scheme for clustering instability is developed based on crossvalidation. The proposed selection criterion's effectiveness is demonstrated on a variety of numerical experiments, and its asymptotic selection consistency is established when the dataset is properly split. Copyright 2010, Oxford University Press.

Suggested Citation

  • Junhui Wang, 2010. "Consistent selection of the number of clusters via crossvalidation," Biometrika, Biometrika Trust, vol. 97(4), pages 893-904.
    • Handle: RePEc:oup:biomet:v:97:y:2010:i:4:p:893-904
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/biomet/asq061
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Julian Rossbroich & Jeffrey Durieux & Tom F. Wilderjans, 2022. "Model Selection Strategies for Determining the Optimal Number of Overlapping Clusters in Additive Overlapping Partitional Clustering," Journal of Classification, Springer;The Classification Society, vol. 39(2), pages 264-301, July.
    2. Chakraborty, Saptarshi & Paul, Debolina & Das, Swagatam, 2020. "Hierarchical clustering with optimal transport," Statistics & Probability Letters, Elsevier, vol. 163(C).
    3. Jingnan Zhang & Chengye Li & Junhui Wang, 2023. "A stochastic block Ising model for multi‐layer networks with inter‐layer dependence," Biometrics, The International Biometric Society, vol. 79(4), pages 3564-3573, December.
    4. Vincent Audigier & Ndèye Niang, 2023. "Clustering with missing data: which equivalent for Rubin’s rules?," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(3), pages 623-657, September.
    5. Will Wei Sun & Xingye Qiao & Guang Cheng, 2016. "Stabilized Nearest Neighbor Classifier and its Statistical Properties," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(515), pages 1254-1265, July.
    6. Han Yu & Brian Chapman & Arianna Di Florio & Ellen Eischen & David Gotz & Mathews Jacob & Rachael Hageman Blair, 2019. "Bootstrapping estimates of stability for clusters, observations and model selection," Computational Statistics, Springer, vol. 34(1), pages 349-372, March.
    7. Victoria Gregory & Guido Menzio & David G. Wiczer, 2021. "The Alpha Beta Gamma of the Labor Market," NBER Working Papers 28663, National Bureau of Economic Research, Inc.
    8. Kensuke Tanioka & Hiroshi Yadohisa, 2019. "Simultaneous Method of Orthogonal Non-metric Non-negative Matrix Factorization and Constrained Non-hierarchical Clustering," Journal of Classification, Springer;The Classification Society, vol. 36(1), pages 73-93, April.
    9. Jonas M. B. Haslbeck & Dirk U. Wulff, 2020. "Estimating the number of clusters via a corrected clustering instability," Computational Statistics, Springer, vol. 35(4), pages 1879-1894, December.
    10. Paul, Biplab & De, Shyamal K. & Ghosh, Anil K., 2022. "Some clustering-based exact distribution-free k-sample tests applicable to high dimension, low sample size data," Journal of Multivariate Analysis, Elsevier, vol. 190(C).
    11. Xianpeng Mao & Yuning Yang, 2022. "Several approximation algorithms for sparse best rank-1 approximation to higher-order tensors," Journal of Global Optimization, Springer, vol. 84(1), pages 229-253, September.
    12. Peter Radchenko & Gourab Mukherjee, 2017. "Convex clustering via l 1 fusion penalization," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(5), pages 1527-1546, November.
    13. Lim, Alejandro & Chiang, Chin-Tsang & Teng, Jen-Chieh, 2021. "Estimating robot strengths with application to selection of alliance members in FIRST robotics competitions," Computational Statistics & Data Analysis, Elsevier, vol. 158(C).
    14. Minjie Wang & Tianyi Yao & Genevera I. Allen, 2023. "Supervised convex clustering," Biometrics, The International Biometric Society, vol. 79(4), pages 3846-3858, December.
    15. Jiangtao Duan & Wei Gao & Hao Qu & Hon Keung Tony, 2019. "Subspace Clustering for Panel Data with Interactive Effects," Papers 1909.09928, arXiv.org, revised Feb 2021.
    16. Rozmus Dorota, 2020. "Clustering Poland Among Eu Countries in Terms of a Sustainable Development Level in the Light of Various Cluster Stability Measures," Folia Oeconomica Stetinensia, Sciendo, vol. 20(1), pages 319-340, June.
    17. Tsubasa Ito & Shonosuke Sugasawa, 2023. "Grouped generalized estimating equations for longitudinal data analysis," Biometrics, The International Biometric Society, vol. 79(3), pages 1868-1879, September.
    18. Mao, Xianpeng & Yang, Yuning, 2022. "Best sparse rank-1 approximation to higher-order tensors via a truncated exponential induced regularizer," Applied Mathematics and Computation, Elsevier, vol. 433(C).
    19. Wang, Junhui & Fang, Yixin, 2013. "Analysis of presence-only data via semi-supervised learning approaches," Computational Statistics & Data Analysis, Elsevier, vol. 59(C), pages 134-143.
    20. Fang, Yixin & Wang, Junhui, 2012. "Selection of the number of clusters via the bootstrap method," Computational Statistics & Data Analysis, Elsevier, vol. 56(3), pages 468-477.
    21. Dario Cottafava & Giulia Sonetti & Paolo Gambino & Andrea Tartaglino, 2018. "Explorative Multidimensional Analysis for Energy Efficiency: DataViz versus Clustering Algorithms," Energies, MDPI, vol. 11(5), pages 1-18, May.
    22. Zhang, Yingying & Wang, Huixia Judy & Zhu, Zhongyi, 2019. "Quantile-regression-based clustering for panel data," Journal of Econometrics, Elsevier, vol. 213(1), pages 54-67.
    23. Yoshikazu Terada, 2014. "Strong Consistency of Reduced K-means Clustering," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 913-931, December.
    24. Zhang, Tonglin & Lin, Ge, 2021. "Generalized k-means in GLMs with applications to the outbreak of COVID-19 in the United States," Computational Statistics & Data Analysis, Elsevier, vol. 159(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:biomet:v:97:y:2010:i:4:p:893-904. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/biomet .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.