IDEAS home Printed from https://ideas.repec.org/a/bla/jinfst/v74y2023i11p1307-1324.html
   My bibliography  Save this article

Do you see what I see? Measuring the semantic differences in image‐recognition services' outputs

Author

Listed:
  • Anton Berg
  • Matti Nelimarkka

Abstract

As scholars increasingly undertake large‐scale analysis of visual materials, advanced computational tools show promise for informing that process. One technique in the toolbox is image recognition, made readily accessible via Google Vision AI, Microsoft Azure Computer Vision, and Amazon's Rekognition service. However, concerns about such issues as bias factors and low reliability have led to warnings against research employing it. A systematic study of cross‐service label agreement concretized such issues: using eight datasets, spanning professionally produced and user‐generated images, the work showed that image‐recognition services disagree on the most suitable labels for images. Beyond supporting caveats expressed in prior literature, the report articulates two mitigation strategies, both involving the use of multiple image‐recognition services: Highly explorative research could include all the labels, accepting noisier but less restrictive analysis output. Alternatively, scholars may employ word‐embedding‐based approaches to identify concepts that are similar enough for their purposes, then focus on those labels filtered in.

Suggested Citation

  • Anton Berg & Matti Nelimarkka, 2023. "Do you see what I see? Measuring the semantic differences in image‐recognition services' outputs," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(11), pages 1307-1324, November.
    • Handle: RePEc:bla:jinfst:v:74:y:2023:i:11:p:1307-1324
    DOI: 10.1002/asi.24827
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/asi.24827
    Download Restriction: no
    File URL: https://libkey.io/10.1002/asi.24827?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Denny, Matthew J. & Spirling, Arthur, 2018. "Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It," Political Analysis, Cambridge University Press, vol. 26(2), pages 168-189, April.
    2. Mike Thelwall & Olga Goriunova & Farida Vis & Simon Faulkner & Anne Burns & Jim Aulich & Amalia Mas-Bleda & Emma Stuart & Francesco D'Orazio, 2016. "Chatting through pictures? A classification of images tweeted in one week in the UK and USA," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(11), pages 2575-2586, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mircea Popa, 2025. "Modelling policy action using natural language processing: evidence for a long-run increase in policy activism in the UK," Journal of Computational Social Science, Springer, vol. 8(2), pages 1-51, May.
    2. Camilla Salvatore & Silvia Biffignandi & Annamaria Bianchi, 2024. "Augmenting business statistics information by combining traditional data with textual data: a composite indicator approach," METRON, Springer;Sapienza Università di Roma, vol. 82(1), pages 71-91, April.
    3. Tyler Andrew Scott & Nicola Ulibarri & Omar Perez Figueroa, 2020. "NEPA and National Trends in Federal Infrastructure Siting in the United States," Review of Policy Research, Policy Studies Organization, vol. 37(5), pages 605-633, September.
    4. Latifi, Albina & Naboka-Krell, Viktoriia & Tillmann, Peter & Winker, Peter, 2024. "Fiscal policy in the Bundestag: Textual analysis and macroeconomic effects," European Economic Review, Elsevier, vol. 168(C).
    5. Bastiaan Bruinsma & Moa Johansson, 2024. "Finding the structure of parliamentary motions in the Swedish Riksdag 1971–2015," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(4), pages 3275-3301, August.
    6. Seraphine F. Maerz & Carsten Q. Schneider, 2020. "Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 517-545, April.
    7. repec:osf:socarx:yfzsh_v1 is not listed on IDEAS
    8. Kim, Da Yeon & Kim, Sang Yong, 2022. "The impact of customer-generated evaluation information on sales in online platform-based markets," Journal of Retailing and Consumer Services, Elsevier, vol. 68(C).
    9. Wang, Yan & Luo, Ting, 2023. "Politicizing for the idol: China’s idol fandom nationalism in pandemic," LSE Research Online Documents on Economics 117741, London School of Economics and Political Science, LSE Library.
    10. Javier De la Hoz-M & Mª José Fernández-Gómez & Susana Mendes, 2021. "LDAShiny: An R Package for Exploratory Review of Scientific Literature Based on a Bayesian Probabilistic Model and Machine Learning Tools," Mathematics, MDPI, vol. 9(14), pages 1-21, July.
    11. Yeomans, Michael, 2021. "A concrete example of construct construction in natural language," Organizational Behavior and Human Decision Processes, Elsevier, vol. 162(C), pages 81-94.
    12. Miklos Sebők & Zoltán Kacsuk & Ákos Máté, 2022. "The (real) need for a human touch: testing a human–machine hybrid topic classification workflow on a New York Times corpus," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(5), pages 3621-3643, October.
    13. Ai-Hsuan Chiang & Silvana Trimi & Tun-Chih Kou, 2024. "Critical Factors for Implementing Smart Manufacturing: A Supply Chain Perspective," Sustainability, MDPI, vol. 16(22), pages 1-21, November.
    14. Andres Algaba & David Ardia & Keven Bluteau & Samuel Borms & Kris Boudt, 2020. "Econometrics Meets Sentiment: An Overview Of Methodology And Applications," Journal of Economic Surveys, Wiley Blackwell, vol. 34(3), pages 512-547, July.
    15. Mohamed M. Mostafa, 2023. "A one-hundred-year structural topic modeling analysis of the knowledge structure of international management research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3905-3935, August.
    16. Kaiser, Carolin & Ahuvia, Aaron & Rauschnabel, Philipp A. & Wimble, Matt, 2020. "Social media monitoring: What can marketers learn from Facebook brand photos?," Journal of Business Research, Elsevier, vol. 117(C), pages 707-717.
    17. Josephine Lukito & Jason Greenfield & Yunkang Yang & Ross Dahlke & Megan A. Brown & Rebecca Lewis & Bin Chen, 2024. "Audio-as-Data Tools: Replicating Computational Data Processing," Media and Communication, Cogitatio Press, vol. 12.
    18. Iasmin Goes, 2023. "Examining the effect of IMF conditionality on natural resource policy," Economics and Politics, Wiley Blackwell, vol. 35(1), pages 227-285, March.
    19. Camilla Salvatore, 2023. "Inference with non-probability samples and survey data integration: a science mapping study," METRON, Springer;Sapienza Università di Roma, vol. 81(1), pages 83-107, April.
    20. Justyna Klejdysz & Robin L. Lumsdaine, 2023. "Shifts in ECB Communication: A Textual Analysis of the Press Conference," International Journal of Central Banking, International Journal of Central Banking, vol. 19(2), pages 473-542, June.
    21. Tim Schatto-Eckrodt & Robin Janzik & Felix Reer & Svenja Boberg & Thorsten Quandt, 2020. "A Computational Approach to Analyzing the Twitter Debate on Gaming Disorder," Media and Communication, Cogitatio Press, vol. 8(3), pages 205-218.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jinfst:v:74:y:2023:i:11:p:1307-1324. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.