IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v7y2013i1p138-144.html
   My bibliography  Save this article

First in, best dressed: The presence of order-effect bias in journal ranking surveys

Author

Listed:
  • Serenko, Alexander
  • Bontis, Nick

Abstract

The purpose of this study is to test for the presence of order-effect bias in journal ranking surveys. Data were obtained from 379 active knowledge management and intellectual capital researchers who rated 25 journals on a 7-point scale. Five different versions of the survey instrument were utilized. Consistent with the cognitive elaboration model, the satisficing theory, and the Gricean maxim of orderliness, order-effect bias was observed in journal ranking surveys. Journals that appear in the beginning of the ranking list delivered to survey respondents consistently receive higher scores than journals at the end of the list. Overall, the position of the journal in the list explains over 10% of its score. Therefore, authors of journal ranking studies are recommended to use multiple versions of the survey instrument with randomized journal orders.

Suggested Citation

  • Serenko, Alexander & Bontis, Nick, 2013. "First in, best dressed: The presence of order-effect bias in journal ranking surveys," Journal of Informetrics, Elsevier, vol. 7(1), pages 138-144.
    • Handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144
    DOI: 10.1016/j.joi.2012.10.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157712000879
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2012.10.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    2. Julian Diaz & Roy T. Black & Joseph Rabianski, 1996. "A Note on Ranking Real Estate Research Journals," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 24(4), pages 551-563, December.
    3. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    4. Clyde W. Holsapple, 2008. "A publication power approach for identifying premier information systems journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(2), pages 166-185, January.
    5. Michael E. Whitman & Anthony R. Hendrickson & Anthony M. Townsend, 1999. "Research Commentary. Academic Rewards for Teaching, Research, and Service: Data and Discourse," Information Systems Research, INFORMS, vol. 10(2), pages 99-109, June.
    6. Serenko, Alexander & Bontis, Nick, 2011. "What's familiar is excellent: The impact of exposure effect on perceived journal quality," Journal of Informetrics, Elsevier, vol. 5(1), pages 219-223.
    7. Blaise Cronin & Lokman I. Meho, 2008. "Applying the author affiliation index to library and information science journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1861-1865, September.
    8. L. Egghe, 2010. "The distribution of the uncitedness factor and its functional relation with the impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 689-695, June.
    9. Israel, Glenn D. & Taylor, C. L., 1990. "Can response order bias evaluations?," Evaluation and Program Planning, Elsevier, vol. 13(4), pages 365-371, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mieko Fujisawa & Kazuhisa Takemura & Yukihiko Funaki & Nobuyuki Uto & Ryo Takahashi, 2020. "An Experimental Study of the Effect of Energy Label Design on the Correct Evaluation of Buildings' Energy Performance and Promotion of Energy Saving," International Real Estate Review, Asian Real Estate Society, vol. 23(1), pages 733-762.
    2. Domingo Docampo & Vicente Safón, 2021. "Journal ratings: a paper affiliation methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8063-8090, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    2. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    3. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    4. Kishen Iyengar & Venugopal Balijepally, 2015. "Ranking journals using the dominance hierarchy procedure: an illustration with IS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 5-23, January.
    5. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    6. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    7. Leah Melani Christian & Nicholas L. Parsons & Don A. Dillman, 2009. "Designing Scalar Questions for Web Surveys," Sociological Methods & Research, , vol. 37(3), pages 393-425, February.
    8. Tim Klopries, 2018. "Discussion of “Working from Home—What is the Effect on Employees’ Effort?”," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 70(1), pages 57-62, February.
    9. Denter, Philipp, 2013. "A theory of communication in political campaigns," Economics Working Paper Series 1302, University of St. Gallen, School of Economics and Political Science.
    10. Kaile Gong, 2023. "The influence of discipline consistency between papers and published journals on citations: an analysis of Chinese papers in three social science disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 3129-3146, May.
    11. Ryazanova, Olga & McNamara, Peter & Aguinis, Herman, 2017. "Research performance as a quality signal in international labor markets: Visibility of business schools worldwide through a global research performance system," Journal of World Business, Elsevier, vol. 52(6), pages 831-841.
    12. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    13. Denter, Philipp, 2020. "Campaign contests," European Economic Review, Elsevier, vol. 127(C).
    14. Müller, Sune Dueholm & Mathiassen, Lars & Balshøj, Hans Henrik, 2008. "Organizational Change Perspectives on Software Process Improvement," Informatics Research Group Working Papers I-2008-02, University of Aarhus, Aarhus School of Business, Department of Business Studies.
    15. repec:ner:leuven:urn:hdl:123456789/327130 is not listed on IDEAS
    16. Timothy D. Fry & Joan M. Donohue, 2014. "Exploring the Author Affiliation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1647-1667, March.
    17. Jerome K. Vanclay & Lutz Bornmann, 2012. "Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 751-771, June.
    18. Fang Xu & Wenbin Liu & Ronald Rousseau, 2015. "Introducing sub-impact factor (SIF-) sequences and an aggregated SIF-indicator for journal ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1577-1593, February.
    19. Daniel E. O'Leary, 2009. "Business School Research: Measuring Value Contribution Through Citations of Journals in Patents," Interfaces, INFORMS, vol. 39(6), pages 516-526, December.
    20. Antonia Ferrer-Sapena & Susana Díaz-Novillo & Enrique A. Sánchez-Pérez, 2017. "Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p -Variations," Publications, MDPI, vol. 5(3), pages 1-14, September.
    21. A. Abrizah & A. Noorhidawati & A. N. Zainab, 2015. "LIS journals categorization in the Journal Citation Report: a stated preference study," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1083-1099, February.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.