IDEAS home Printed from https://ideas.repec.org/a/bla/jinfst/v74y2023i11p1307-1324.html
   My bibliography  Save this article

Do you see what I see? Measuring the semantic differences in image‐recognition services' outputs

Author

Listed:
  • Anton Berg
  • Matti Nelimarkka

Abstract

As scholars increasingly undertake large‐scale analysis of visual materials, advanced computational tools show promise for informing that process. One technique in the toolbox is image recognition, made readily accessible via Google Vision AI, Microsoft Azure Computer Vision, and Amazon's Rekognition service. However, concerns about such issues as bias factors and low reliability have led to warnings against research employing it. A systematic study of cross‐service label agreement concretized such issues: using eight datasets, spanning professionally produced and user‐generated images, the work showed that image‐recognition services disagree on the most suitable labels for images. Beyond supporting caveats expressed in prior literature, the report articulates two mitigation strategies, both involving the use of multiple image‐recognition services: Highly explorative research could include all the labels, accepting noisier but less restrictive analysis output. Alternatively, scholars may employ word‐embedding‐based approaches to identify concepts that are similar enough for their purposes, then focus on those labels filtered in.

Suggested Citation

  • Anton Berg & Matti Nelimarkka, 2023. "Do you see what I see? Measuring the semantic differences in image‐recognition services' outputs," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(11), pages 1307-1324, November.
  • Handle: RePEc:bla:jinfst:v:74:y:2023:i:11:p:1307-1324
    DOI: 10.1002/asi.24827
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/asi.24827
    Download Restriction: no

    File URL: https://libkey.io/10.1002/asi.24827?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mike Thelwall & Olga Goriunova & Farida Vis & Simon Faulkner & Anne Burns & Jim Aulich & Amalia Mas-Bleda & Emma Stuart & Francesco D'Orazio, 2016. "Chatting through pictures? A classification of images tweeted in one week in the UK and USA," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(11), pages 2575-2586, November.
    2. Denny, Matthew J. & Spirling, Arthur, 2018. "Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It," Political Analysis, Cambridge University Press, vol. 26(2), pages 168-189, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mohamed M. Mostafa, 2023. "A one-hundred-year structural topic modeling analysis of the knowledge structure of international management research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3905-3935, August.
    2. Latifi, Albina & Naboka-Krell, Viktoriia & Tillmann, Peter & Winker, Peter, 2024. "Fiscal policy in the Bundestag: Textual analysis and macroeconomic effects," European Economic Review, Elsevier, vol. 168(C).
    3. Karina Shyrokykh & Max Girnyk & Lisa Dellmuth, 2023. "Short text classification with machine learning in the social sciences: The case of climate change on Twitter," PLOS ONE, Public Library of Science, vol. 18(9), pages 1-26, September.
    4. Purwoko Haryadi Santoso & Edi Istiyono & Haryanto & Wahyu Hidayatulloh, 2022. "Thematic Analysis of Indonesian Physics Education Research Literature Using Machine Learning," Data, MDPI, vol. 7(11), pages 1-41, October.
    5. Noailly, Joëlle & Nowzohour, Laura & van den Heuvel, Matthias & Pla, Ireneu, 2024. "Heard the news? Environmental policy and clean investments," Journal of Public Economics, Elsevier, vol. 238(C).
    6. Camilla Salvatore & Silvia Biffignandi & Annamaria Bianchi, 2022. "Corporate Social Responsibility Activities Through Twitter: From Topic Model Analysis to Indexes Measuring Communication Characteristics," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(3), pages 1217-1248, December.
    7. Jason Anastasopoulos & George J. Borjas & Gavin G. Cook & Michael Lachanski, 2018. "Job Vacancies, the Beveridge Curve, and Supply Shocks: The Frequency and Content of Help-Wanted Ads in Pre- and Post-Mariel Miami," NBER Working Papers 24580, National Bureau of Economic Research, Inc.
    8. repec:osf:socarx:yfzsh_v1 is not listed on IDEAS
    9. W. Benedikt Schmal, 2024. "Academic Knowledge: Does it Reflect the Combinatorial Growth of Technology?," Papers 2409.20282, arXiv.org.
    10. Lorien Stice-Lawrence, 2022. "Practical issues to consider when working with big data," Review of Accounting Studies, Springer, vol. 27(3), pages 1117-1124, September.
    11. Camilla Salvatore & Silvia Biffignandi & Annamaria Bianchi, 2024. "Augmenting business statistics information by combining traditional data with textual data: a composite indicator approach," METRON, Springer;Sapienza Università di Roma, vol. 82(1), pages 71-91, April.
    12. Tyler Andrew Scott & Nicola Ulibarri & Omar Perez Figueroa, 2020. "NEPA and National Trends in Federal Infrastructure Siting in the United States," Review of Policy Research, Policy Studies Organization, vol. 37(5), pages 605-633, September.
    13. Seraphine F. Maerz & Carsten Q. Schneider, 2020. "Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 517-545, April.
    14. Kim, Da Yeon & Kim, Sang Yong, 2022. "The impact of customer-generated evaluation information on sales in online platform-based markets," Journal of Retailing and Consumer Services, Elsevier, vol. 68(C).
    15. Iasmin Goes, 2023. "Examining the effect of IMF conditionality on natural resource policy," Economics and Politics, Wiley Blackwell, vol. 35(1), pages 227-285, March.
    16. Erkan Gunes & Christoffer Koch Florczak, 2025. "Replacing or enhancing the human coder? Multiclass classification of policy documents with large language models," Journal of Computational Social Science, Springer, vol. 8(2), pages 1-20, May.
    17. Shah Jahan Miah & Huy Quan Vu & Damminda Alahakoon, 2022. "A social media analytics perspective for human‐oriented smart city planning and management," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(1), pages 119-135, January.
    18. Khandaker Mohammad Mohi Uddin & Hasibul Hamim & Mst Nishat Tasnim Mim & Arnisha Akhter & Md Ashraf Uddin, 2024. "Machine learning and deep learning-based approach to categorize Bengali comments on social networks using fused dataset," PLOS ONE, Public Library of Science, vol. 19(10), pages 1-35, October.
    19. Mircea Popa, 2025. "Modelling policy action using natural language processing: evidence for a long-run increase in policy activism in the UK," Journal of Computational Social Science, Springer, vol. 8(2), pages 1-51, May.
    20. Yeomans, Michael, 2021. "A concrete example of construct construction in natural language," Organizational Behavior and Human Decision Processes, Elsevier, vol. 162(C), pages 81-94.
    21. Ai-Hsuan Chiang & Silvana Trimi & Tun-Chih Kou, 2024. "Critical Factors for Implementing Smart Manufacturing: A Supply Chain Perspective," Sustainability, MDPI, vol. 16(22), pages 1-21, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jinfst:v:74:y:2023:i:11:p:1307-1324. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.