IDEAS home Printed from https://ideas.repec.org/a/sae/joudef/v22y2025i4p429-441.html

Efficient estimation of the number of clusters for high-dimension data

Author

Listed:
  • Spiridon Kasapis
  • Geng Zhang
  • Jonathon M Smereka
  • Nickolas Vlahopoulos

Abstract

The exponential growth of digital image data has given rise to the need of efficient content management and retrieval tools. Currently, there is a lack of tools for processing the collected unlabeled data in a schematic manner. K -means is one of the most widely used clustering methods and has been applied in a variety of fields, one of them being image sorting. Although a useful tool for image management, the K -means method is heavily influenced by initializations, the most important one being the need to know the number of clusters a priori. A number of different methods have been proposed for identifying the correct number of clusters for K -means, one of them being the variance ratio criterion (VRC). Despite its popularity, the VRC method comes with two very important shortcomings: it only yields good results when the data dimensionality is low and it does not scale well for a high number of clusters, making it very difficult to use in computer vision applications. We propose an extension to the VRC method that works for increased cluster number and high-dimensionality data sets and therefore is fit for image data sets.

Suggested Citation

  • Spiridon Kasapis & Geng Zhang & Jonathon M Smereka & Nickolas Vlahopoulos, 2025. "Efficient estimation of the number of clusters for high-dimension data," The Journal of Defense Modeling and Simulation, , vol. 22(4), pages 429-441, October.
    • Handle: RePEc:sae:joudef:v:22:y:2025:i:4:p:429-441
    DOI: 10.1177/15485129231214569
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/15485129231214569
    Download Restriction: no
    File URL: https://libkey.io/10.1177/15485129231214569?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mardia, K. V., 1988. "Multi-dimensional multivariate Gaussian Markov random fields with application to image processing," Journal of Multivariate Analysis, Elsevier, vol. 24(2), pages 265-284, February.
    2. Atif Ishtiaq & Sheeraz Ahmed & Muhammad Fahad Khan & Farhan Aadil & Muazzam Maqsood & Salabat Khan, 2019. "Intelligent clustering using moth flame optimizer for vehicular ad hoc networks," International Journal of Distributed Sensor Networks, , vol. 15(1), pages 15501477188, January.
    3. Robert Tibshirani & Guenther Walther & Trevor Hastie, 2001. "Estimating the number of clusters in a data set via the gap statistic," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(2), pages 411-423.
    4. Fangzheng Xie & Yanxun Xu, 2020. "Bayesian Repulsive Gaussian Mixture Model," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(529), pages 187-203, January.
    5. Hamparsum Bozdogan, 1987. "Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions," Psychometrika, Springer;The Psychometric Society, vol. 52(3), pages 345-370, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Julian Rossbroich & Jeffrey Durieux & Tom F. Wilderjans, 2022. "Model Selection Strategies for Determining the Optimal Number of Overlapping Clusters in Additive Overlapping Partitional Clustering," Journal of Classification, Springer;The Classification Society, vol. 39(2), pages 264-301, July.
    2. repec:plo:pone00:0178915 is not listed on IDEAS
    3. Ioana Gutu & Daniela Tatiana Agheorghiesei & Alexandru Tugui, 2023. "Assessment of a Workforce Sustainability Tool through Leadership and Digitalization," IJERPH, MDPI, vol. 20(2), pages 1-30, January.
    4. Thiemo Fetzer & Samuel Marden, 2017. "Take What You Can: Property Rights, Contestability and Conflict," Economic Journal, Royal Economic Society, vol. 0(601), pages 757-783, May.
    5. Daniela Andreini & Diego Rinallo & Giuseppe Pedeliento & Mara Bergamaschi, 2017. "Brands and Religion in the Secularized Marketplace and Workplace: Insights from the Case of an Italian Hospital Renamed After a Roman Catholic Pope," Journal of Business Ethics, Springer, vol. 141(3), pages 529-550, March.
    6. Daniel Agness & Travis Baseler & Sylvain Chassang & Pascaline Dupas & Erik Snowberg, 2025. "Valuing the Time of the Self-Employed," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 92(6), pages 3471-3503.
    7. Khanh Duong, 2024. "Is meritocracy just? New evidence from Boolean analysis and Machine learning," Journal of Computational Social Science, Springer, vol. 7(2), pages 1795-1821, October.
    8. Batool, Fatima & Hennig, Christian, 2021. "Clustering with the Average Silhouette Width," Computational Statistics & Data Analysis, Elsevier, vol. 158(C).
    9. Nicoleta Serban & Huijing Jiang, 2012. "Multilevel Functional Clustering Analysis," Biometrics, The International Biometric Society, vol. 68(3), pages 805-814, September.
    10. Gao, Lily (Xuehui) & Melero-Polo, Iguácel & Sese, F. Javier, 2025. "The role of customer experience dimensions in expanding customer–firm relationships: A customer expansion journey approach," Journal of Retailing, Elsevier, vol. 101(3), pages 493-517.
    11. S. A. Abu Bakar & Saralees Nadarajah & Z. A. Absl Kamarul Adzhar, 2018. "Loss modeling using Burr mixtures," Empirical Economics, Springer, vol. 54(4), pages 1503-1516, June.
    12. Byrd, T. A. & Marshall, T. E., 1997. "Relating information technology investment to organizational performance: a causal model analysis," Omega, Elsevier, vol. 25(1), pages 43-56, February.
    13. José J. Quinlan & Fernando A. Quintana & Garritt L. Page, 2021. "On a class of repulsive mixture models," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 30(2), pages 445-461, June.
    14. Anahita Nodehi & Mousa Golalizadeh & Mehdi Maadooliat & Claudio Agostinelli, 2025. "Torus Probabilistic Principal Component Analysis," Journal of Classification, Springer;The Classification Society, vol. 42(2), pages 435-456, July.
    15. Orietta Nicolis & Jean Paul Maidana & Fabian Contreras & Danilo Leal, 2024. "Analyzing the Impact of COVID-19 on Economic Sustainability: A Clustering Approach," Sustainability, MDPI, vol. 16(4), pages 1-30, February.
    16. Herbert Hoijtink & Meinte Vollema, 2003. "Contemporary Extensions of the Rasch Model," Quality & Quantity: International Journal of Methodology, Springer, vol. 37(3), pages 263-276, August.
    17. Jaewoong Yun, 2023. "Strategies for Improving the Sustainability of Fare-Free Policy for the Elderly through Preferences by Travel Modes," Sustainability, MDPI, vol. 15(20), pages 1-14, October.
    18. Malerba, Martino E. & Connolly, Sean R. & Heimann, Kirsten, 2015. "An experimentally validated nitrate–ammonium–phytoplankton model including effects of starvation length and ammonium inhibition on nitrate uptake," Ecological Modelling, Elsevier, vol. 317(C), pages 30-40.
    19. Li, Pai-Ling & Chiou, Jeng-Min, 2011. "Identifying cluster number for subspace projected functional data clustering," Computational Statistics & Data Analysis, Elsevier, vol. 55(6), pages 2090-2103, June.
    20. Aline Riboli Marasca & Maurício Scopel Hoffmann & Anelise Reis Gaya & Denise Ruschel Bandeira, 2021. "Subjective Well-Being and Psychopathology Symptoms: Mental Health Profiles and their Relations with Academic Achievement in Brazilian Children," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 14(3), pages 1121-1137, June.
    21. Yaeji Lim & Hee-Seok Oh & Ying Kuen Cheung, 2019. "Multiscale Clustering for Functional Data," Journal of Classification, Springer;The Classification Society, vol. 36(2), pages 368-391, July.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:joudef:v:22:y:2025:i:4:p:429-441. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.