IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v9y2018i1d10.1038_s41467-018-07619-7.html
   My bibliography  Save this article

Why rankings of biomedical image analysis competitions should be interpreted with care

Author

Listed:
  • Lena Maier-Hein

    (German Cancer Research Center (DKFZ))

  • Matthias Eisenmann

    (German Cancer Research Center (DKFZ))

  • Annika Reinke

    (German Cancer Research Center (DKFZ))

  • Sinan Onogur

    (German Cancer Research Center (DKFZ))

  • Marko Stankovic

    (German Cancer Research Center (DKFZ))

  • Patrick Scholz

    (German Cancer Research Center (DKFZ))

  • Tal Arbel

    (McGill University)

  • Hrvoje Bogunovic

    (Medical University Vienna)

  • Andrew P. Bradley

    (Queensland University of Technology)

  • Aaron Carass

    (Johns Hopkins University)

  • Carolin Feldmann

    (German Cancer Research Center (DKFZ))

  • Alejandro F. Frangi

    (The University of Leeds)

  • Peter M. Full

    (German Cancer Research Center (DKFZ))

  • Bram Ginneken

    (Radboud University Center)

  • Allan Hanbury

    (Institute of Information Systems Engineering, TU Wien
    Complexity Science Hub Vienna)

  • Katrin Honauer

    (Heidelberg University)

  • Michal Kozubek

    (Masaryk University)

  • Bennett A. Landman

    (Vanderbilt University)

  • Keno März

    (German Cancer Research Center (DKFZ))

  • Oskar Maier

    (Universität zu Lübeck)

  • Klaus Maier-Hein

    (German Cancer Research Center (DKFZ))

  • Bjoern H. Menze

    (Technical University of Munich)

  • Henning Müller

    (Information System Institute, HES-SO)

  • Peter F. Neher

    (German Cancer Research Center (DKFZ))

  • Wiro Niessen

    (Nuclear Medicine and Medical Informatics, Erasmus MC)

  • Nasir Rajpoot

    (University of Warwick)

  • Gregory C. Sharp

    (Massachusetts General Hospital)

  • Korsuk Sirinukunwattana

    (University of Oxford)

  • Stefanie Speidel

    (National Center for Tumor Diseases Dresden)

  • Christian Stock

    (German Cancer Research Center (DKFZ))

  • Danail Stoyanov

    (University College London)

  • Abdel Aziz Taha

    (Research Studios Austria FG)

  • Fons Sommen

    (Eindhoven University of Technology)

  • Ching-Wei Wang

    (National Taiwan University of Science and Technology)

  • Marc-André Weber

    (University Medical Center Rostock)

  • Guoyan Zheng

    (University of Bern)

  • Pierre Jannin

    (LTSI (Laboratoire Traitement du Signal et de l’Image) - UMR_S 1099)

  • Annette Kopp-Schneider

    (German Cancer Research Center (DKFZ))

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Suggested Citation

  • Lena Maier-Hein & Matthias Eisenmann & Annika Reinke & Sinan Onogur & Marko Stankovic & Patrick Scholz & Tal Arbel & Hrvoje Bogunovic & Andrew P. Bradley & Aaron Carass & Carolin Feldmann & Alejandro , 2018. "Why rankings of biomedical image analysis competitions should be interpreted with care," Nature Communications, Nature, vol. 9(1), pages 1-13, December.
    • Handle: RePEc:nat:natcom:v:9:y:2018:i:1:d:10.1038_s41467-018-07619-7
    DOI: 10.1038/s41467-018-07619-7
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-018-07619-7
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-018-07619-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yashvardhan Jain & Leah L. Godwin & Sripad Joshi & Shriya Mandarapu & Trang Le & Cecilia Lindskog & Emma Lundberg & Katy Börner, 2023. "Segmenting functional tissue units across human organs using community-driven development of generalizable machine learning algorithms," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    2. Maximilian Zenk & Ujjwal Baid & Sarthak Pati & Akis Linardos & Brandon Edwards & Micah Sheller & Patrick Foley & Alejandro Aristizabal & David Zimmerer & Alexey Gruzdev & Jason Martin & Russell T. Shi, 2025. "Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge," Nature Communications, Nature, vol. 16(1), pages 1-20, December.
    3. Ezequiel Rosa & Mauricio Reyes & Sook-Lei Liew & Alexandre Hutton & Roland Wiest & Johannes Kaesmacher & Uta Hanning & Arsany Hakim & Richard Zubal & Waldo Valenzuela & David Robben & Diana M. Sima & , 2025. "DeepISLES: a clinically validated ischemic stroke segmentation model from the ISLES'22 challenge," Nature Communications, Nature, vol. 16(1), pages 1-16, December.
    4. Michela Antonelli & Annika Reinke & Spyridon Bakas & Keyvan Farahani & Annette Kopp-Schneider & Bennett A. Landman & Geert Litjens & Bjoern Menze & Olaf Ronneberger & Ronald M. Summers & Bram Ginneken, 2022. "The Medical Segmentation Decathlon," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    5. Christina Fell & Mahnaz Mohammadi & David Morrison & Ognjen Arandjelovic & Peter Caie & David Harris-Birtill, 2022. "Reproducibility of deep learning in digital pathology whole slide image analysis," PLOS Digital Health, Public Library of Science, vol. 1(12), pages 1-21, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:9:y:2018:i:1:d:10.1038_s41467-018-07619-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.