IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0304807.html
   My bibliography  Save this article

Perceptions and detection of AI use in manuscript preparation for academic journals

Author

Listed:
  • Nir Chemaya
  • Daniel Martin

Abstract

The rapid advances in Generative AI tools have produced both excitement and worry about how AI will impact academic writing. However, little is known about what norms are emerging around AI use in manuscript preparation or how these norms might be enforced. We address both gaps in the literature by conducting a survey of 271 academics about whether it is necessary to report ChatGPT use in manuscript preparation and by running GPT-modified abstracts from 2,716 published papers through a leading AI detection software to see if these detectors can detect different AI uses in manuscript preparation. We find that most academics do not think that using ChatGPT to fix grammar needs to be reported, but detection software did not always draw this distinction, as abstracts for which GPT was used to fix grammar were often flagged as having a high chance of being written by AI. We also find disagreements among academics on whether more substantial use of ChatGPT to rewrite text needs to be reported, and these differences were related to perceptions of ethics, academic role, and English language background. Finally, we found little difference in their perceptions about reporting ChatGPT and research assistant help, but significant differences in reporting perceptions between these sources of assistance and paid proofreading and other AI assistant tools (Grammarly and Word). Our results suggest that there might be challenges in getting authors to report AI use in manuscript preparation because (i) there is not uniform agreement about what uses of AI should be reported and (ii) journals might have trouble enforcing nuanced reporting requirements using AI detection tools.

Suggested Citation

  • Nir Chemaya & Daniel Martin, 2024. "Perceptions and detection of AI use in manuscript preparation for academic journals," PLOS ONE, Public Library of Science, vol. 19(7), pages 1-16, July.
    • Handle: RePEc:plo:pone00:0304807
    DOI: 10.1371/journal.pone.0304807
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0304807
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0304807&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0304807?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bjorkegren, Dan & Blumenstock, Joshua & Knight, Samsun, 2022. "(Machine) Learning What Policies Value," CEPR Discussion Papers 17364, C.E.P.R. Discussion Papers.
    2. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2019. "The Economics of Artificial Intelligence: An Agenda," NBER Books, National Bureau of Economic Research, Inc, number agra-1, March.
    3. Harjit Singh & Avneet Singh, 2023. "ChatGPT: Systematic Review, Applications, and Agenda for Multidisciplinary Research," Journal of Chinese Economic and Business Studies, Taylor & Francis Journals, vol. 21(2), pages 193-212, April.
    4. Drew Fudenberg & Annie Liang, 2019. "Predicting and Understanding Initial Play," American Economic Review, American Economic Association, vol. 109(12), pages 4112-4141, December.
    5. Anja Lambrecht & Catherine Tucker, 2019. "Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads," Management Science, INFORMS, vol. 65(7), pages 2966-2981, July.
    6. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    7. Chen-Fu Chien & Stéphane Dauzère-Pérès & Woonghee Tim Huh & Young Jae Jang & James R. Morrison, 2020. "Artificial intelligence in manufacturing and logistics systems: algorithms, applications, and case studies," International Journal of Production Research, Taylor & Francis Journals, vol. 58(9), pages 2730-2731, May.
    8. David Dranove & Ginger Zhe Jin, 2010. "Quality Disclosure and Certification: Theory and Practice," Journal of Economic Literature, American Economic Association, vol. 48(4), pages 935-963, December.
    9. Sendhil Mullainathan & Jann Spiess, 2017. "Machine Learning: An Applied Econometric Approach," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 87-106, Spring.
    10. Anton Korinek, 2023. "Language Models and Cognitive Automation for Economic Research," NBER Working Papers 30957, National Bureau of Economic Research, Inc.
    11. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nir Chemaya & Daniel Martin, 2023. "Perceptions and Detection of AI Use in Manuscript Preparation for Academic Journals," Papers 2311.14720, arXiv.org, revised Jan 2024.
    2. Felix Chopra & Ingar Haaland, 2023. "Conducting qualitative interviews with AI," CEBI working paper series 23-06, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    3. Gary Charness & Brian Jabarian & John List, 2023. "Generation Next: Experimentation with AI," Artefactual Field Experiments 00777, The Field Experiments Website.
    4. Shoshan, Vered & Hazan, Tamir & Plonsky, Ori, 2023. "BEAST-Net: Learning novel behavioral insights using a neural network adaptation of a behavioral model," OSF Preprints kaeny, Center for Open Science.
    5. Byron Botha & Rulof Burger & Kevin Kotzé & Neil Rankin & Daan Steenkamp, 2023. "Big data forecasting of South African inflation," Empirical Economics, Springer, vol. 65(1), pages 149-188, July.
    6. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    7. Combes, Pierre-Philippe & Gobillon, Laurent & Zylberberg, Yanos, 2022. "Urban economics in a historical perspective: Recovering data with machine learning," Regional Science and Urban Economics, Elsevier, vol. 94(C).
    8. Rama K. Malladi, 2024. "Benchmark Analysis of Machine Learning Methods to Forecast the U.S. Annual Inflation Rate During a High-Decile Inflation Period," Computational Economics, Springer;Society for Computational Economics, vol. 64(1), pages 335-375, July.
    9. Kevin Leyton-Brown & Paul Milgrom & Neil Newman & Ilya Segal, 2024. "Artificial Intelligence and Market Design: Lessons Learned from Radio Spectrum Reallocation," NBER Chapters, in: New Directions in Market Design, National Bureau of Economic Research, Inc.
    10. Capra, C. Monica & Kniesner, Thomas J., 2025. "Daniel Kahneman’s Underappreciated Last Published Paper: Empirical Implications for Benefit-Cost Analysis and a Chat Session Discussion with Bots," IZA Discussion Papers 17841, Institute of Labor Economics (IZA).
    11. Kirshner, Samuel N., 2024. "GPT and CLT: The impact of ChatGPT's level of abstraction on consumer recommendations," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    12. Zengqing Wu & Run Peng & Xu Han & Shuyuan Zheng & Yixin Zhang & Chuan Xiao, 2023. "Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations," Papers 2311.06330, arXiv.org, revised Dec 2023.
    13. repec:osf:osfxxx:udz28_v1 is not listed on IDEAS
    14. Joshua C. Yang & Damian Dailisan & Marcin Korecki & Carina I. Hausladen & Dirk Helbing, 2024. "LLM Voting: Human Choices and AI Collective Decision Making," Papers 2402.01766, arXiv.org, revised Aug 2024.
    15. Majid Majzoubi & Eric Yanfei Zhao, 2023. "Going beyond optimal distinctiveness: Strategic positioning for gaining an audience composition premium," Strategic Management Journal, Wiley Blackwell, vol. 44(3), pages 737-777, March.
    16. Lijia Ma & Xingchen Xu & Yong Tan, 2024. "Crafting Knowledge: Exploring the Creative Mechanisms of Chat-Based Search Engines," Papers 2402.19421, arXiv.org.
    17. Ali Goli & Amandeep Singh, 2023. "Exploring the Influence of Language on Time-Reward Perceptions in Large Language Models: A Study Using GPT-3.5," Papers 2305.02531, arXiv.org, revised Jun 2023.
    18. repec:osf:osfxxx:r3qng_v1 is not listed on IDEAS
    19. Evangelos Katsamakas, 2024. "Business models for the simulation hypothesis," Papers 2404.08991, arXiv.org.
    20. Prithwiraj Choudhury & Dan Wang & Natalie A. Carlson & Tarun Khanna, 2019. "Machine learning approaches to facial and text analysis: Discovering CEO oral communication styles," Strategic Management Journal, Wiley Blackwell, vol. 40(11), pages 1705-1732, November.
    21. Yuan Gao & Dokyun Lee & Gordon Burtch & Sina Fazelpour, 2024. "Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina," Papers 2410.19599, arXiv.org, revised Jan 2025.
    22. Yongtong Shao & Tao Xiong & Minghao Li & Dermot Hayes & Wendong Zhang & Wei Xie, 2021. "China's Missing Pigs: Correcting China's Hog Inventory Data Using a Machine Learning Approach," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(3), pages 1082-1098, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0304807. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.