IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v112y2017i1d10.1007_s11192-017-2284-3.html
   My bibliography  Save this article

Quality assessment of scientific outputs using the BWM

Author

Listed:
  • Negin Salimi

    (Leiden University)

Abstract

Assessing the quality of scientific outputs (i.e. research papers, books and reports) is a challenging issue. Although in practice, the basic quality of scientific outputs is evaluated by committees/peers (peer review) who have general knowledge and competencies. However, their assessment might not comprehensively consider different dimensions of the quality of the scientific outputs. Hence, there is a requirement to evaluate scientific outputs based on some other metrics which cover more aspects of quality after publishing, which is the aim of this study. To reach this aim, first different quality metrics are identified through an extensive literature review. Then a recently developed multi-criteria methodology (best worst method) is used to find the importance of each quality metric. Finally, based on the importance of each quality metric and the data which are collected from Scopus, the quality of research papers published by the members of a university faculty is measured. The proposed model in this paper provides the opportunity to measure quality of research papers not only by considering different aspects of quality, but also by considering the importance of each quality metric. The proposed model can be used for assessing other scientific outputs as well.

Suggested Citation

  • Negin Salimi, 2017. "Quality assessment of scientific outputs using the BWM," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 195-213, July.
    • Handle: RePEc:spr:scient:v:112:y:2017:i:1:d:10.1007_s11192-017-2284-3
    DOI: 10.1007/s11192-017-2284-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-017-2284-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-017-2284-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pauline Mattsson & Patrice Laget & Anna Nilsson & Carl-Johan Sundberg, 2008. "Intra-EU vs. extra-EU scientific co-publication patterns in EU," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(3), pages 555-574, June.
    2. Liwen Vaughan & Debora Shaw, 2005. "Web citation data for impact assessment: A comparison of four science disciplines," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(10), pages 1075-1087, August.
    3. Moed, H. F. & Burger, W. J. M. & Frankfort, J. G. & Van Raan, A. F. J., 1985. "The use of bibliometric data for the measurement of university research performance," Research Policy, Elsevier, vol. 14(3), pages 131-149, June.
    4. Henk F Moed, 2007. "The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review," Science and Public Policy, Oxford University Press, vol. 34(8), pages 575-583, October.
    5. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    6. E. C. M. Noyons & H. F. Moed & A. F. J. Raan, 1999. "Integrating research performance analysis and science mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 591-604, November.
    7. Negin Salimi & Jafar Rezaei, 2016. "Measuring efficiency of university-industry Ph.D. projects using best worst method," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1911-1938, December.
    8. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    9. Abbasi, Alireza & Altmann, Jörn & Hossain, Liaquat, 2011. "Identifying the effects of co-authorship networks on the performance of scholars: A correlation and regression analysis of performance measures and social network analysis measures," Journal of Informetrics, Elsevier, vol. 5(4), pages 594-607.
    10. Walter, Jorge & Lechner, Christoph & Kellermanns, Franz W., 2007. "Knowledge transfer between and within alliance partners: Private versus collective benefits of social capital," Journal of Business Research, Elsevier, vol. 60(7), pages 698-710, July.
    11. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    12. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    13. Li, Eldon Y. & Liao, Chien Hsiang & Yen, Hsiuju Rebecca, 2013. "Co-authorship networks and research impact: A social capital perspective," Research Policy, Elsevier, vol. 42(9), pages 1515-1530.
    14. Hildrun Kretschmer, 2004. "Author productivity and geodesic distance in bibliographic co-authorship networks, and visibility on the Web," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(3), pages 409-420, August.
    15. Bornmann, Lutz & Leydesdorff, Loet, 2013. "The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000," Journal of Informetrics, Elsevier, vol. 7(2), pages 286-291.
    16. Rezaei, Jafar, 2016. "Best-worst multi-criteria decision-making method: Some properties and a linear model," Omega, Elsevier, vol. 64(C), pages 126-130.
    17. Rezaei, Jafar, 2015. "Best-worst multi-criteria decision-making method," Omega, Elsevier, vol. 53(C), pages 49-57.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. G. Yoganandan & M. Vasan, 2022. "Evaluating the quality of scientific research papers in entrepreneurship," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(5), pages 3013-3027, October.
    2. Geerten Van de Kaa & Daniel Scholten & Jafar Rezaei & Christine Milchram, 2017. "The Battle between Battery and Fuel Cell Powered Electric Vehicles: A BWM Approach," Energies, MDPI, vol. 10(11), pages 1-13, October.
    3. Linhong Xu & Kun Ding & Yuan Lin & Chunbo Zhang, 2023. "Does citation polarity help evaluate the quality of academic papers?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 4065-4087, July.
    4. Cristina López-Duarte & Marta M. Vidal-Suárez & Belén González-Díaz, 2019. "Cross-national distance and international business: an analysis of the most influential recent models," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 173-208, October.
    5. Govindan, Kannan & Shankar, K. Madan & Kannan, Devika, 2020. "Achieving sustainable development goals through identifying and analyzing barriers to industrial sharing economy: A framework development," International Journal of Production Economics, Elsevier, vol. 227(C).
    6. Salimi, Negin & Rezaei, Jafar, 2018. "Evaluating firms’ R&D performance using best worst method," Evaluation and Program Planning, Elsevier, vol. 66(C), pages 147-155.
    7. Željko Stević & Irena Đalić & Dragan Pamučar & Zdravko Nunić & Slavko Vesković & Marko Vasiljević & Ilija Tanackov, 2019. "A new hybrid model for quality assessment of scientific conferences based on Rough BWM and SERVQUAL," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 1-30, April.
    8. Mi, Xiaomei & Tang, Ming & Liao, Huchang & Shen, Wenjing & Lev, Benjamin, 2019. "The state-of-the-art survey on integrations and applications of the best worst method in decision making: Why, what, what for and what's next?," Omega, Elsevier, vol. 87(C), pages 205-225.
    9. Mohammadi, Majid & Rezaei, Jafar, 2020. "Bayesian best-worst method: A probabilistic group decision making model," Omega, Elsevier, vol. 96(C).
    10. Yonghe Lu & Jiayi Luo & Ying Xiao & Hou Zhu, 2021. "Text representation model of scientific papers based on fusing multi-viewpoint information and its quality assessment," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6937-6963, August.
    11. Mohammad Khalilzadeh & Laleh Katoueizadeh & Edmundas Kazimieras Zavadskas, 2020. "Risk identification and prioritization in banking projects of payment service provider companies: an empirical study," Frontiers of Business Research in China, Springer, vol. 14(1), pages 1-27, December.
    12. Ieva Meidute-Kavaliauskiene & Vida Davidaviciene & Gencay Karakaya & Shahryar Ghorbani, 2021. "The Measurement of Organizational Social Media Integration Impact on Financial and Innovative Performance: An Integrated Model," Sustainability, MDPI, vol. 13(18), pages 1-16, September.
    13. Rezaei, Jafar & van Roekel, Wilco S. & Tavasszy, Lori, 2018. "Measuring the relative importance of the logistics performance index indicators using Best Worst Method," Transport Policy, Elsevier, vol. 68(C), pages 158-169.
    14. Mališa Žižović & Dragan Pamučar & Goran Ćirović & Miodrag M. Žižović & Boža D. Miljković, 2020. "A Model for Determining Weight Coefficients by Forming a Non-Decreasing Series at Criteria Significance Levels (NDSL)," Mathematics, MDPI, vol. 8(5), pages 1-18, May.
    15. Badri Ahmadi, Hadi & Kusi-Sarpong, Simonov & Rezaei, Jafar, 2017. "Assessing the social sustainability of supply chains using Best Worst Method," Resources, Conservation & Recycling, Elsevier, vol. 126(C), pages 99-106.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    2. Vinayak, & Raghuvanshi, Adarsh & kshitij, Avinash, 2023. "Signatures of capacity development through research collaborations in artificial intelligence and machine learning," Journal of Informetrics, Elsevier, vol. 17(1).
    3. Salimi, Negin & Rezaei, Jafar, 2018. "Evaluating firms’ R&D performance using best worst method," Evaluation and Program Planning, Elsevier, vol. 66(C), pages 147-155.
    4. Hamzeh Soltanali & Mehdi Khojastehpour & Siamak Kheybari, 2023. "Evaluating the critical success factors for maintenance management in agro-industries using multi-criteria decision-making techniques," Operations Management Research, Springer, vol. 16(2), pages 949-968, June.
    5. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    6. Javid Nafari & Alireza Arab & Sina Ghaffari, 2017. "Through the Looking Glass: Analysis of Factors Influencing Iranian Student’s Study Abroad Motivations and Destination Choice," SAGE Open, , vol. 7(2), pages 21582440177, June.
    7. Sabrina Petersohn & Thomas Heinze, 2018. "Professionalization of bibliometric research assessment. Insights from the history of the Leiden Centre for Science and Technology Studies (CWTS)," Science and Public Policy, Oxford University Press, vol. 45(4), pages 565-578.
    8. Lutz Bornmann, 2015. "Interrater reliability and convergent validity of F1000Prime peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(12), pages 2415-2426, December.
    9. Gholamreza Haseli & Reza Sheikh & Jianqiang Wang & Hana Tomaskova & Erfan Babaee Tirkolaee, 2021. "A Novel Approach for Group Decision Making Based on the Best–Worst Method (G-BWM): Application to Supply Chain Management," Mathematics, MDPI, vol. 9(16), pages 1-20, August.
    10. Geerten Van de Kaa & Daniel Scholten & Jafar Rezaei & Christine Milchram, 2017. "The Battle between Battery and Fuel Cell Powered Electric Vehicles: A BWM Approach," Energies, MDPI, vol. 10(11), pages 1-13, October.
    11. Li, Eldon Y. & Liao, Chien Hsiang & Yen, Hsiuju Rebecca, 2013. "Co-authorship networks and research impact: A social capital perspective," Research Policy, Elsevier, vol. 42(9), pages 1515-1530.
    12. Mališa Žižović & Dragan Pamučar & Goran Ćirović & Miodrag M. Žižović & Boža D. Miljković, 2020. "A Model for Determining Weight Coefficients by Forming a Non-Decreasing Series at Criteria Significance Levels (NDSL)," Mathematics, MDPI, vol. 8(5), pages 1-18, May.
    13. Mohammadi, Majid & Rezaei, Jafar, 2020. "Bayesian best-worst method: A probabilistic group decision making model," Omega, Elsevier, vol. 96(C).
    14. Chi-Yen Yin & Hsiao-Hsin Chang, 2022. "What Is the Link between Strategic Innovation and Organizational Sustainability? Historical Review and Bibliometric Analytics," Sustainability, MDPI, vol. 14(11), pages 1-23, June.
    15. Zohreh Zahedi & Rodrigo Costas & Paul Wouters, 2014. "How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1491-1513, November.
    16. Andersen, Jens Peter, 2017. "An empirical and theoretical critique of the Euclidean index," Journal of Informetrics, Elsevier, vol. 11(2), pages 455-465.
    17. Mi, Xiaomei & Tang, Ming & Liao, Huchang & Shen, Wenjing & Lev, Benjamin, 2019. "The state-of-the-art survey on integrations and applications of the best worst method in decision making: Why, what, what for and what's next?," Omega, Elsevier, vol. 87(C), pages 205-225.
    18. Huseyin Kocak & Atalay Caglar & Gulin Zeynep Oztas, 2018. "Euclidean Best–Worst Method and Its Application," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 17(05), pages 1587-1605, September.
    19. Shih-Chia Chang & Ming-Tsang Lu & Mei-Jen Chen & Li-Hua Huang, 2021. "Evaluating the Application of CSR in the High-Tech Industry during the COVID-19 Pandemic," Mathematics, MDPI, vol. 9(15), pages 1-16, July.
    20. Gupta, Himanshu, 2018. "Evaluating service quality of airline industry using hybrid best worst method and VIKOR," Journal of Air Transport Management, Elsevier, vol. 68(C), pages 35-47.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:112:y:2017:i:1:d:10.1007_s11192-017-2284-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.