IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v8y2014i1p77-88.html
   My bibliography  Save this article

Measuring journal performance for multidisciplinary research: An efficiency perspective

Author

Listed:
  • Lee, Hakyeon
  • Shin, Juneseuk

Abstract

One of the flaws of the journal impact factor (IF) is that it cannot be used to compare journals from different fields or multidisciplinary journals because the IF differs significantly across research fields. This study proposes a new measure of journal performance that captures field-different citation characteristics. We view journal performance from the perspective of the efficiency of a journal's citation generation process. Together with the conventional variables used in calculating the IF, the number of articles as an input and the number of total citations as an output, we additionally consider the two field-different factors, citation density and citation dynamics, as inputs. We also separately capture the contribution of external citations and self-citations and incorporate their relative importance in measuring journal performance. To accommodate multiple inputs and outputs whose relationships are unknown, this study employs data envelopment analysis (DEA), a multi-factor productivity model for measuring the relative efficiency of decision-making units without any assumption of a production function. The resulting efficiency score, called DEA-IF, can then be used for the comparative evaluation of multidisciplinary journals’ performance. A case study example of industrial engineering journals is provided to illustrate how to measure DEA-IF and its usefulness.

Suggested Citation

  • Lee, Hakyeon & Shin, Juneseuk, 2014. "Measuring journal performance for multidisciplinary research: An efficiency perspective," Journal of Informetrics, Elsevier, vol. 8(1), pages 77-88.
    • Handle: RePEc:eee:infome:v:8:y:2014:i:1:p:77-88
    DOI: 10.1016/j.joi.2013.10.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157713000813
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2013.10.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Johnes, Geraint & Johnes, Jill, 1993. "Measuring the Research Performance of UK Economics Departments: An Application of Data Envelopment Analysis," Oxford Economic Papers, Oxford University Press, vol. 45(2), pages 332-347, April.
    2. Cherchye, L. & Abeele, P. Vanden, 2005. "On research efficiency: A micro-analysis of Dutch university research in Economics and Business Management," Research Policy, Elsevier, vol. 34(4), pages 495-516, May.
    3. Per Andersen & Niels Christian Petersen, 1993. "A Procedure for Ranking Efficient Units in Data Envelopment Analysis," Management Science, INFORMS, vol. 39(10), pages 1261-1264, October.
    4. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    5. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    6. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    7. Dorta-González, P. & Dorta-González, M.I., 2013. "Impact maturity times and citation time windows: The 2-year maximum journal impact factor," Journal of Informetrics, Elsevier, vol. 7(3), pages 593-602.
    8. R. Allen & A. Athanassopoulos & R.G. Dyson & E. Thanassoulis, 1997. "Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions," Annals of Operations Research, Springer, vol. 73(0), pages 13-34, October.
    9. Petridis, Konstantinos & Malesios, Chrisovalantis & Arabatzis, Garyfallos & Thanassoulis, Emmanuel, 2013. "Efficiency analysis of forestry journals: Suggestions for improving journals’ quality," Journal of Informetrics, Elsevier, vol. 7(2), pages 505-521.
    10. Liu, John S. & Lu, Wen-Min, 2010. "DEA and ranking with the network-based approach: a case of R&D performance," Omega, Elsevier, vol. 38(6), pages 453-464, December.
    11. Hsu, Fang-Ming & Hsueh, Chao-Chih, 2009. "Measuring relative efficiency of government-sponsored R&D projects: A three-stage approach," Evaluation and Program Planning, Elsevier, vol. 32(2), pages 178-186, May.
    12. Thompson, Russell G. & Langemeier, Larry N. & Lee, Chih-Tah & Lee, Euntaik & Thrall, Robert M., 1990. "The role of multiplier bounds in efficiency analysis with application to Kansas farming," Journal of Econometrics, Elsevier, vol. 46(1-2), pages 93-108.
    13. Wagner, Caroline S. & Roessner, J. David & Bobb, Kamau & Klein, Julie Thompson & Boyack, Kevin W. & Keyton, Joann & Rafols, Ismael & Börner, Katy, 2011. "Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature," Journal of Informetrics, Elsevier, vol. 5(1), pages 14-26.
    14. Lee, Hakyeon & Park, Yongtae & Choi, Hoogon, 2009. "Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach," European Journal of Operational Research, Elsevier, vol. 196(3), pages 847-855, August.
    15. Wang, Eric C. & Huang, Weichiao, 2007. "Relative efficiency of R&D activities: A cross-country study accounting for environmental factors in the DEA approach," Research Policy, Elsevier, vol. 36(2), pages 260-273, March.
    16. Korhonen, Pekka & Tainio, Risto & Wallenius, Jyrki, 2001. "Value efficiency analysis of academic research," European Journal of Operational Research, Elsevier, vol. 130(1), pages 121-132, April.
    17. Seema Sharma & V. J. Thomas, 2008. "Inter-country R&D efficiency analysis: An application of data envelopment analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(3), pages 483-501, September.
    18. Groot, Tom & Garcia-Valderrama, Teresa, 2006. "Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs," Research Policy, Elsevier, vol. 35(9), pages 1362-1376, November.
    19. R. D. Banker & A. Charnes & W. W. Cooper, 1984. "Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis," Management Science, INFORMS, vol. 30(9), pages 1078-1092, September.
    20. Eilat, Harel & Golany, Boaz & Shtub, Avraham, 2006. "Constructing and evaluating balanced portfolios of R&D projects with interactions: A DEA based methodology," European Journal of Operational Research, Elsevier, vol. 172(3), pages 1018-1039, August.
    21. Franceschet, Massimo, 2010. "Journal influence factors," Journal of Informetrics, Elsevier, vol. 4(3), pages 239-248.
    22. Vanclay, Jerome K., 2013. "Factors affecting citation rates in environmental science," Journal of Informetrics, Elsevier, vol. 7(2), pages 265-271.
    23. Thed N. van Leeuwen & Henk F. Moed, 2002. "Development and application of journal impact measures in the Dutch science system," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 249-266, February.
    24. William W. Cooper & Lawrence M. Seiford & Kaoru Tone, 2007. "Data Envelopment Analysis," Springer Books, Springer, edition 0, number 978-0-387-45283-8, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Liu, Hui-hui & Yang, Guo-liang & Liu, Xiao-xiao & Song, Yao-yao, 2020. "R&D performance assessment of industrial enterprises in China: A two-stage DEA approach," Socio-Economic Planning Sciences, Elsevier, vol. 71(C).
    2. Xu, Nuo & Kasimov, Ikboljon & Wang, Yanan, 2022. "Unlocking private investment as a new determinant of green finance for renewable development in China," Renewable Energy, Elsevier, vol. 198(C), pages 1121-1130.
    3. Lee, Seonghee & Lee, Hakyeon, 2015. "Measuring and comparing the R&D performance of government research institutes: A bottom-up data envelopment analysis approach," Journal of Informetrics, Elsevier, vol. 9(4), pages 942-953.
    4. Ennas, Gianfranco & Di Guardo, Maria Chiara, 2015. "Features of top-rated gold open access journals: An analysis of the scopus database," Journal of Informetrics, Elsevier, vol. 9(1), pages 79-89.
    5. Chen, Kun & Ren, Xian-tong & Yang, Guo-liang, 2021. "A novel approach for assessing academic journals: Application of integer DEA model for management science and operations research field," Journal of Informetrics, Elsevier, vol. 15(3).
    6. Yang, Guoliang & Ahlgren, Per & Yang, Liying & Rousseau, Ronald & Ding, Jielan, 2016. "Using multi-level frontiers in DEA models to grade countries/territories," Journal of Informetrics, Elsevier, vol. 10(1), pages 238-253.
    7. Hung, Chia-Liang, 2017. "Social networks, technology ties, and gatekeeper functionality: Implications for the performance management of R&D projects," Research Policy, Elsevier, vol. 46(1), pages 305-315.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lee, Seonghee & Lee, Hakyeon, 2015. "Measuring and comparing the R&D performance of government research institutes: A bottom-up data envelopment analysis approach," Journal of Informetrics, Elsevier, vol. 9(4), pages 942-953.
    2. Khoshnevis, Pegah & Teirlinck, Peter, 2018. "Performance evaluation of R&D active firms," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 16-28.
    3. Liu, John S. & Lu, Wen-Min, 2010. "DEA and ranking with the network-based approach: a case of R&D performance," Omega, Elsevier, vol. 38(6), pages 453-464, December.
    4. Yang, Guoliang & Ahlgren, Per & Yang, Liying & Rousseau, Ronald & Ding, Jielan, 2016. "Using multi-level frontiers in DEA models to grade countries/territories," Journal of Informetrics, Elsevier, vol. 10(1), pages 238-253.
    5. Lee, Hakyeon & Park, Yongtae & Choi, Hoogon, 2009. "Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach," European Journal of Operational Research, Elsevier, vol. 196(3), pages 847-855, August.
    6. Junhee Bae & Yanghon Chung & Hyesoo Ko, 2021. "Analysis of efficiency in public research activities in terms of knowledge spillover: focusing on earthquake R&D accomplishments," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 108(2), pages 2249-2264, September.
    7. Vladimír Holý & Karel Šafr, 2018. "Are economically advanced countries more efficient in basic and applied research?," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 26(4), pages 933-950, December.
    8. Jaehun Park & Joonyoung Kim & Si-Il Sung, 2017. "Performance Evaluation of Research and Business Development: A Case Study of Korean Public Organizations," Sustainability, MDPI, vol. 9(12), pages 1-16, December.
    9. Jung Ho Park & Kwangsoo Shin, 2018. "Efficiency of Government-Sponsored R&D Projects: A Metafrontier DEA Approach," Sustainability, MDPI, vol. 10(7), pages 1-17, July.
    10. Tavana, Madjid & Ebrahimnejad, Ali & Santos-Arteaga, Francisco J. & Mansourzadeh, Seyed Mehdi & Matin, Reza Kazemi, 2018. "A hybrid DEA-MOLP model for public school assessment and closure decision in the City of Philadelphia," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 70-89.
    11. Svetlana V. Ratner & Svetlana A. Balashova & Andrey V. Lychev, 2022. "The Efficiency of National Innovation Systems in Post-Soviet Countries: DEA-Based Approach," Mathematics, MDPI, vol. 10(19), pages 1-23, October.
    12. André Klevenhusen & Jonas Coelho & Leo Warszawski & Jorge Moreira & Peter Wanke & João J. Ferreira, 2021. "Innovation Efficiency in OECD Countries: a Non-parametric Approach," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 12(3), pages 1064-1078, September.
    13. Jianping Liu & Kai Lu & Shixiong Cheng, 2018. "International R&D Spillovers and Innovation Efficiency," Sustainability, MDPI, vol. 10(11), pages 1-23, October.
    14. Hung, Chia-Liang, 2017. "Social networks, technology ties, and gatekeeper functionality: Implications for the performance management of R&D projects," Research Policy, Elsevier, vol. 46(1), pages 305-315.
    15. Ebrahimi, Bohlool & Dhamotharan, Lalitha & Ghasemi, Mohammad Reza & Charles, Vincent, 2022. "A cross-inefficiency approach based on the deviation variables framework," Omega, Elsevier, vol. 111(C).
    16. Mai, Nhat Chi, 2015. "Efficiency of the banking system in Vietnam under financial liberalization," OSF Preprints qsf6d, Center for Open Science.
    17. Svetlana Ratner & Andrey Lychev & Aleksei Rozhnov & Igor Lobanov, 2021. "Efficiency Evaluation of Regional Environmental Management Systems in Russia Using Data Envelopment Analysis," Mathematics, MDPI, vol. 9(18), pages 1-21, September.
    18. Kerstin Pull & Birgit Pferdmenges & Uschi Backes-Gellner, 2017. "Do Research Training Groups Operate at Optimal Size?," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 18(2), pages 129-145, May.
    19. Chen Kaihua & Kou Mingting, 2014. "Staged efficiency and its determinants of regional innovation systems: a two-step analytical procedure," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 52(2), pages 627-657, March.
    20. Rafael Benítez & Vicente Coll-Serrano & Vicente J. Bolós, 2021. "deaR-Shiny: An Interactive Web App for Data Envelopment Analysis," Sustainability, MDPI, vol. 13(12), pages 1-19, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:8:y:2014:i:1:p:77-88. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.