IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0310321.html

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

Author

Listed:
  • Jong-Wook Ban
  • Lucy Abel
  • Richard Stevens
  • Rafael Perera

Abstract

Background: External validation studies create evidence about a clinical prediction rule’s (CPR’s) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential. Methods: A forward citation search of the Framingham Wilson CHD risk rule’s derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule. Results: We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data. Discussion: Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule’s generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.

Suggested Citation

  • Jong-Wook Ban & Lucy Abel & Richard Stevens & Rafael Perera, 2024. "Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review," PLOS ONE, Public Library of Science, vol. 19(9), pages 1-21, September.
    • Handle: RePEc:plo:pone00:0310321
    DOI: 10.1371/journal.pone.0310321
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0310321
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0310321&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0310321?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:plo:pone00:0179102 is not listed on IDEAS
    2. Alberto Martín-Martín & Mike Thelwall & Enrique Orduna-Malea & Emilio Delgado López-Cózar, 2021. "Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 871-906, January.
    3. Alberto Martín-Martín & Mike Thelwall & Enrique Orduna-Malea & Emilio Delgado López-Cózar, 2021. "Correction to: Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 907-908, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ramón A. Feenstra & Emilio Delgado López-Cózar, 2022. "Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2085-2103, April.
    2. Xu, Shuo & Wang, Congcong & An, Xin & Deng, Yunkang & Liu, Jianhua, 2025. "Do OpenCitations and Dimensions serve as an alternative to Web of Science for calculating disruption indexes?," Journal of Informetrics, Elsevier, vol. 19(3).
    3. Mehdi Toloo & Rouhollah Khodabandelou & Amar Oukil, 2022. "A Comprehensive Bibliometric Analysis of Fractional Programming (1965–2020)," Mathematics, MDPI, vol. 10(11), pages 1-21, May.
    4. Dušan Nikolić & Dragan Ivanović & Lidija Ivanović, 2024. "An open-source tool for merging data from multiple citation databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4573-4595, July.
    5. Mike Thelwall & Stephen Pinfield, 2024. "The accuracy of field classifications for journals in Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(2), pages 1097-1117, February.
    6. Anting Wang & Mohd Nizam Osman & Megat Al-Imran Yasin & Nurul Ain Mohd Hasan & Ying Cui, 2024. "Tracing Evolution and Communication Dynamics in Chinese Independent Documentary Films (2012-2022): A Systematic Review of Genre, Censorship, Culture, and Distribution," Studies in Media and Communication, Redfame publishing, vol. 12(1), pages 368-381, March.
    7. Elena Andriollo & Alberto Caimo & Laura Secco & Elena Pisani, 2021. "Collaborations in Environmental Initiatives for an Effective “Adaptive Governance” of Social–Ecological Systems: What Existing Literature Suggests," Sustainability, MDPI, vol. 13(15), pages 1-29, July.
    8. repec:dar:wpaper:132320 is not listed on IDEAS
    9. Do Khac, Lilian Tai & Leyer, Michael, 2026. "Towards an integrative model of organizational human-AI collaboration: A semi-systematic review of the current state of the art," Technology in Society, Elsevier, vol. 84(C).
    10. Enrique Orduña-Malea & Núria Bautista-Puig, 2024. "Research assessment under debate: disentangling the interest around the DORA declaration on Twitter," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 537-559, January.
    11. Kelly Gerakoudi-Ventouri, 2022. "Review of studies of blockchain technology effects on the shipping industry," Journal of Shipping and Trade, Springer, vol. 7(1), pages 1-18, December.
    12. Luiza Loredana Năstase, 2025. "Sustainable Education and University Students’ Well-Being in the Digital Age: A Mixed-Methods Study on Problematic Smartphone Use," Sustainability, MDPI, vol. 17(13), pages 1-34, June.
    13. Safoora Pitsi & Jon Billsberry & Mary Barrett, 2024. "A Bibliometric Review of Research on Intelligence in Leadership Studies," FIIB Business Review, , vol. 13(5), pages 528-541, October.
    14. Yang Ding & Fernando Moreira, 2025. "Funding and productivity: Does winning grants affect the scientific productivity of recipients? Evidence from the social sciences and economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1831-1870, March.
    15. DIODATO Dario, 2024. "Handbook of Economic Complexity for Policy," JRC Research Reports JRC138666, Joint Research Centre.
    16. Toluwase Asubiaro & Sodiq Onaolapo & David Mills, 2024. "Regional disparities in Web of Science and Scopus journal coverage," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(3), pages 1469-1491, March.
    17. Zhang, Yang & Wang, Yang & Du, Haifeng & Havlin, Shlomo, 2024. "Delayed citation impact of interdisciplinary research," Journal of Informetrics, Elsevier, vol. 18(1).
    18. Alejandro Valencia-Arias & Juana Ramírez Dávila & Wilmer Londoño-Celis & Lucia Palacios-Moya & Julio Leyrer Hernández & Erica Agudelo-Ceballos & Hernán Uribe-Bedoya, 2024. "Research Trends in the Use of the Internet of Things in Sustainability Practices: A Systematic Review," Sustainability, MDPI, vol. 16(7), pages 1-23, March.
    19. Nezha Mejjad & Marzia Rovere, 2021. "Understanding the Impacts of Blue Economy Growth on Deep-Sea Ecosystem Services," Sustainability, MDPI, vol. 13(22), pages 1-26, November.
    20. Ruben Tessmann & Ralf Elbert, 2022. "Multi-sided platforms in competitive B2B networks with varying governmental influence – a taxonomy of Port and Cargo Community System business models," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(2), pages 829-872, June.
    21. Hui Li, 2025. "Global or regional: the hidden truth behind ShanghaiRanking’s global university ranking by the subject of Law," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(1), pages 515-530, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0310321. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.