IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1007276.html
   My bibliography  Save this article

Benchmarking network propagation methods for disease gene identification

Author

Listed:
  • Sergio Picart-Armada
  • Steven J Barrett
  • David R Willé
  • Alexandre Perera-Lluna
  • Alex Gutteridge
  • Benoit H Dessailly

Abstract

In-silico identification of potential target genes for disease is an essential aspect of drug target discovery. Recent studies suggest that successful targets can be found through by leveraging genetic, genomic and protein interaction information. Here, we systematically tested the ability of 12 varied algorithms, based on network propagation, to identify genes that have been targeted by any drug, on gene-disease data from 22 common non-cancerous diseases in OpenTargets. We considered two biological networks, six performance metrics and compared two types of input gene-disease association scores. The impact of the design factors in performance was quantified through additive explanatory models. Standard cross-validation led to over-optimistic performance estimates due to the presence of protein complexes. In order to obtain realistic estimates, we introduced two novel protein complex-aware cross-validation schemes. When seeding biological networks with known drug targets, machine learning and diffusion-based methods found around 2-4 true targets within the top 20 suggestions. Seeding the networks with genes associated to disease by genetics decreased performance below 1 true hit on average. The use of a larger network, although noisier, improved overall performance. We conclude that diffusion-based prioritisers and machine learning applied to diffusion-based features are suited for drug discovery in practice and improve over simpler neighbour-voting methods. We also demonstrate the large impact of choosing an adequate validation strategy and the definition of seed disease genes.Author summary: The use of biological network data has proven its effectiveness in many areas from computational biology. Networks consist of nodes, usually genes or proteins, and edges that connect pairs of nodes, representing information such as physical interactions, regulatory roles or co-occurrence. In order to find new candidate nodes for a given biological property, the so-called network propagation algorithms start from the set of known nodes with that property and leverage the connections from the biological network to make predictions. Here, we assess the performance of several network propagation algorithms to find sensible gene targets for 22 common non-cancerous diseases, i.e. those that have been found promising enough to start the clinical trials with any compound. We focus on obtaining performance metrics that reflect a practical scenario in drug development where only a small set of genes can be essayed. We found that the presence of protein complexes biased the performance estimates, leading to over-optimistic conclusions, and introduced two novel strategies to address it. Our results support that network propagation is still a viable approach to find drug targets, but that special care needs to be put on the validation strategy. Algorithms benefitted from the use of a larger -although noisier- network and of direct evidence data, rather than indirect genetic associations to disease.

Suggested Citation

  • Sergio Picart-Armada & Steven J Barrett & David R Willé & Alexandre Perera-Lluna & Alex Gutteridge & Benoit H Dessailly, 2019. "Benchmarking network propagation methods for disease gene identification," PLOS Computational Biology, Public Library of Science, vol. 15(9), pages 1-24, September.
  • Handle: RePEc:plo:pcbi00:1007276
    DOI: 10.1371/journal.pcbi.1007276
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007276
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1007276&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1007276?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lori E. Dodd & Margaret S. Pepe, 2003. "Partial AUC Estimation and Regression," Biometrics, The International Biometric Society, vol. 59(3), pages 614-623, September.
    2. Sergio Picart-Armada & Francesc Fernández-Albert & Maria Vinaixa & Miguel A Rodríguez & Suvi Aivio & Travis H Stracker & Oscar Yanes & Alexandre Perera-Lluna, 2017. "Null diffusion-based enrichment for metabolomics data," PLOS ONE, Public Library of Science, vol. 12(12), pages 1-21, December.
    3. Takaya Saito & Marc Rehmsmeier, 2015. "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-21, March.
    4. Karatzoglou, Alexandros & Smola, Alexandros & Hornik, Kurt & Zeileis, Achim, 2004. "kernlab - An S4 Package for Kernel Methods in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 11(i09).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tsukioka, Yasutomo & Yanagi, Junya & Takada, Teruko, 2018. "Investor sentiment extracted from internet stock message boards and IPO puzzles," International Review of Economics & Finance, Elsevier, vol. 56(C), pages 205-217.
    2. Daniel J. Luckett & Eric B. Laber & Samer S. El‐Kamary & Cheng Fan & Ravi Jhaveri & Charles M. Perou & Fatma M. Shebl & Michael R. Kosorok, 2021. "Receiver operating characteristic curves and confidence bands for support vector machines," Biometrics, The International Biometric Society, vol. 77(4), pages 1422-1430, December.
    3. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    4. Grabisch, Michel & Kojadinovic, Ivan & Meyer, Patrick, 2008. "A review of methods for capacity identification in Choquet integral based multi-attribute utility theory: Applications of the Kappalab R package," European Journal of Operational Research, Elsevier, vol. 186(2), pages 766-785, April.
    5. Jie-Huei Wang & Cheng-Yu Liu & You-Ruei Min & Zih-Han Wu & Po-Lin Hou, 2024. "Cancer Diagnosis by Gene-Environment Interactions via Combination of SMOTE-Tomek and Overlapped Group Screening Approaches with Application to Imbalanced TCGA Clinical and Genomic Data," Mathematics, MDPI, vol. 12(14), pages 1-24, July.
    6. Le, Hong Hanh & Viviani, Jean-Laurent, 2018. "Predicting bank failure: An improvement by implementing a machine-learning approach to classical financial ratios," Research in International Business and Finance, Elsevier, vol. 44(C), pages 16-25.
    7. Peterson, A. Townsend & Papeş, Monica & Soberón, Jorge, 2008. "Rethinking receiver operating characteristic analysis applications in ecological niche modeling," Ecological Modelling, Elsevier, vol. 213(1), pages 63-72.
    8. João Chang Junior & Fábio Binuesa & Luiz Fernando Caneo & Aida Luiza Ribeiro Turquetto & Elisandra Cristina Trevisan Calvo Arita & Aline Cristina Barbosa & Alfredo Manoel da Silva Fernandes & Evelinda, 2020. "Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study," PLOS ONE, Public Library of Science, vol. 15(9), pages 1-21, September.
    9. Margaret Sullivan Pepe & Tianxi Cai, 2004. "The Analysis of Placement Values for Evaluating Discriminatory Measures," Biometrics, The International Biometric Society, vol. 60(2), pages 528-535, June.
    10. Bellotti, Anthony & Brigo, Damiano & Gambetti, Paolo & Vrins, Frédéric, 2021. "Forecasting recovery rates on non-performing loans with machine learning," International Journal of Forecasting, Elsevier, vol. 37(1), pages 428-444.
    11. Riza, Lala Septem & Bergmeir, Christoph & Herrera, Francisco & Benítez, José M., 2015. "frbs: Fuzzy Rule-Based Systems for Classification and Regression in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 65(i06).
    12. Karin Wolffhechel & Amanda C Hahn & Hanne Jarmer & Claire I Fisher & Benedict C Jones & Lisa M DeBruine, 2015. "Testing the Utility of a Data-Driven Approach for Assessing BMI from Face Images," PLOS ONE, Public Library of Science, vol. 10(10), pages 1-10, October.
    13. Arthur De Sá Ferreira & Ney Meziat-Filho & Ana Paula Antunes Ferreira, 2021. "Double threshold receiver operating characteristic plot for three-modal continuous predictors," Computational Statistics, Springer, vol. 36(3), pages 2231-2245, September.
    14. Fan, Xudong & Wang, Xiaowei & Zhang, Xijin & ASCE Xiong (Bill) Yu, P.E.F., 2022. "Machine learning based water pipe failure prediction: The effects of engineering, geology, climate and socio-economic factors," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    15. Zhang, Han, 2021. "How Using Machine Learning Classification as a Variable in Regression Leads to Attenuation Bias and What to Do About It," SocArXiv 453jk, Center for Open Science.
    16. Masabho P Milali & Samson S Kiware & Nicodem J Govella & Fredros Okumu & Naveen Bansal & Serdar Bozdag & Jacques D Charlwood & Marta F Maia & Sheila B Ogoma & Floyd E Dowell & George F Corliss & Maggy, 2020. "An autoencoder and artificial neural network-based method to estimate parity status of wild mosquitoes from near-infrared spectra," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-16, June.
    17. Daniel R Jeske, 2018. "Metrics Used When Evaluating the Performance of Statistical Classifiers," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 8(1), pages 7-9, August.
    18. Juliet Chebet Moso & Stéphane Cormier & Cyril de Runz & Hacène Fouchal & John Mwangi Wandeto, 2021. "Anomaly Detection on Data Streams for Smart Agriculture," Agriculture, MDPI, vol. 11(11), pages 1-17, November.
    19. Andrea S Martinez-Vernon & James A Covington & Ramesh P Arasaradnam & Siavash Esfahani & Nicola O’Connell & Ioannis Kyrou & Richard S Savage, 2018. "An improved machine learning pipeline for urinary volatiles disease detection: Diagnosing diabetes," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-20, September.
    20. Kajal Lahiri & Cheng Yang, 2023. "ROC and PRC Approaches to Evaluate Recession Forecasts," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 19(2), pages 119-148, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1007276. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.