IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v219y2025ics0040162525003063.html

Determinants of firms' decisions to block AI crawlers: Strategic insights from the BEDC framework

Author

Listed:
  • Cheng, Chiang-Yu

Abstract

The rise of AI crawlers has intensified firms' concerns over digital asset protection and competitive visibility. This study develops the BEDC Framework—encompassing Business Competition, Engineering Constraints, Domain Authority, and Commerce & Monetization—to conceptualize strategic determinants of AI crawler governance. An empirical analysis of 100 top-ranking global websites reveals significant variation in crawler restriction practices, highlighting the interplay between strategic imperatives and governance behaviors. The findings advance understanding of firm-centric technology governance in the context of automated data extraction, offering insights into how organizations strategically balance protection, accessibility, and monetization.

Suggested Citation

  • Cheng, Chiang-Yu, 2025. "Determinants of firms' decisions to block AI crawlers: Strategic insights from the BEDC framework," Technological Forecasting and Social Change, Elsevier, vol. 219(C).
    • Handle: RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525003063
    DOI: 10.1016/j.techfore.2025.124275
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162525003063
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2025.124275?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Gawer, Annabelle, 2014. "Bridging differing perspectives on technological platforms: Toward an integrative framework," Research Policy, Elsevier, vol. 43(7), pages 1239-1249.
    2. Joan Ofulue & Morad Benyoucef, 2024. "Data monetization: insights from a technology-enabled literature review and research agenda," Management Review Quarterly, Springer, vol. 74(2), pages 521-565, June.
    3. Erik Brynjolfsson & Lorin M. Hitt, 2000. "Beyond Computation: Information Technology, Organizational Transformation and Business Performance," Journal of Economic Perspectives, American Economic Association, vol. 14(4), pages 23-48, Fall.
    4. Geoffrey G. Parker & Marshall W. Van Alstyne, 2005. "Two-Sided Network Effects: A Theory of Information Product Design," Management Science, INFORMS, vol. 51(10), pages 1494-1504, October.
    5. Thomas R. Eisenmann & Geoffrey Parker & Marshall Van Alstyne, 2009. "Opening Platforms: How, When and Why?," Chapters, in: Annabelle Gawer (ed.), Platforms, Markets and Innovation, chapter 6, Edward Elgar Publishing.
    6. Jean-Charles Rochet & Jean Tirole, 2003. "Platform Competition in Two-Sided Markets," Journal of the European Economic Association, MIT Press, vol. 1(4), pages 990-1029, June.
    7. Kevin Boudreau, 2010. "Open Platform Strategies and Innovation: Granting Access vs. Devolving Control," Management Science, INFORMS, vol. 56(10), pages 1849-1872, October.
    8. Rodolphe Durand & Robert M. Grant & Tammy L. Madsen & David P. McIntyre & Arati Srinivasan, 2017. "Networks, platforms, and strategy: Emerging views and next steps," Strategic Management Journal, Wiley Blackwell, vol. 38(1), pages 141-160, January.
    9. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769, December.
    10. Kalyan Talluri & Garrett van Ryzin, 2004. "Revenue Management Under a General Discrete Choice Model of Consumer Behavior," Management Science, INFORMS, vol. 50(1), pages 15-33, January.
    11. Hal R. Varian, 2014. "Big Data: New Tricks for Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 3-28, Spring.
    12. Dominik K. Kanbach & Louisa Heiduk & Georg Blueher & Maximilian Schreiter & Alexander Lahmann, 2024. "The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective," Review of Managerial Science, Springer, vol. 18(4), pages 1189-1220, April.
    13. Sakiru Adebola Solarin, 2025. "Convergence of tourism market diversification: evidence from a new indicator based on Herfindahl–Hirschman index," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(1), pages 663-681, February.
    14. Nelson Granados & Alok Gupta & Robert J. Kauffman, 2012. "Online and Offline Demand and Price Elasticities: Evidence from the Air Travel Industry," Information Systems Research, INFORMS, vol. 23(1), pages 164-181, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tavalaei, M. Mahdi, 2020. "Waiting time in two-sided platforms: The case of the airport industry," Technological Forecasting and Social Change, Elsevier, vol. 159(C).
    2. Shi, Xianwei & Liang, Xingkun & Luo, Yining, 2023. "Unpacking the intellectual structure of ecosystem research in innovation studies," Research Policy, Elsevier, vol. 52(6).
    3. Jingtao Yi & Jinqiu He & Lihong Yang, 2019. "Platform heterogeneity, platform governance and complementors’ product performance: an empirical study of the mobile application industry," Frontiers of Business Research in China, Springer, vol. 13(1), pages 1-20, December.
    4. Liu, He & Li, Xuerong & Wang, Shouyang, 2021. "A bibliometric analysis of 30 years of platform research: Developing the research agenda for platforms, the associated technologies and social impacts," Technological Forecasting and Social Change, Elsevier, vol. 169(C).
    5. Mosterd, Lars & Sobota, Vladimir C.M. & van de Kaa, Geerten & Ding, Aaron Yi & de Reuver, Mark, 2021. "Context dependent trade-offs around platform-to-platform openness: The case of the Internet of Things," Technovation, Elsevier, vol. 108(C).
    6. Jørgen Veisdal, 2020. "The dynamics of entry for digital platforms in two-sided markets: a multi-case study," Electronic Markets, Springer;IIM University of St. Gallen, vol. 30(3), pages 539-556, September.
    7. Jan Frederic Nerbel & Markus Kreutzer, 2023. "Digital platform ecosystems in flux: From proprietary digital platforms to wide-spanning ecosystems," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-20, December.
    8. Jacobides, Michael G. & Cennamo, Carmelo & Gawer, Annabelle, 2024. "Externalities and complementarities in platforms and ecosystems: From structural solutions to endogenous failures," Research Policy, Elsevier, vol. 53(1).
    9. Roma, Paolo & Vasi, Maria, 2019. "Diversification and performance in the mobile app market: The role of the platform ecosystem," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 123-139.
    10. Jens Foerderer & Thomas Kude & Sunil Mithas & Armin Heinzl, 2018. "Does Platform Owner’s Entry Crowd Out Innovation? Evidence from Google Photos," Information Systems Research, INFORMS, vol. 29(2), pages 444-460, June.
    11. Burcu Tan & Edward G. Anderson, Jr. & Geoffrey G. Parker, 2020. "Platform Pricing and Investment to Drive Third-Party Value Creation in Two-Sided Networks," Information Systems Research, INFORMS, vol. 31(1), pages 217-239, March.
    12. Gawer, Annabelle, 2014. "Bridging differing perspectives on technological platforms: Toward an integrative framework," Research Policy, Elsevier, vol. 43(7), pages 1239-1249.
    13. Xing Wan & Javier Cenamor & Geoffrey Parker & Marshall Van Alstyne, 2017. "Unraveling Platform Strategies: A Review from an Organizational Ambidexterity Perspective," Sustainability, MDPI, vol. 9(5), pages 1-18, May.
    14. van de Kaa, Geerten & Papachristos, George & de Bruijn, Hans, 2019. "The governance of platform development processes: A metaphor and a simulation model," Technological Forecasting and Social Change, Elsevier, vol. 138(C), pages 190-203.
    15. Zeng, Jing & Khan, Zaheer & De Silva, Muthu, 2019. "The emergence of multi-sided platform MNEs: Internalization theory and networks," International Business Review, Elsevier, vol. 28(6), pages 1-1.
    16. Vladimir C. M. Sobota & Mark de Reuver & Amir Pooyan Afghari & Geerten van de Kaa, 2026. "Strategic openness: Category variety, boundary resources, and exclusive content as drivers of complementor participation," Electronic Markets, Springer;IIM University of St. Gallen, vol. 36(1), pages 1-19, December.
    17. Andreas Hein & Maximilian Schreieck & Tobias Riasanow & David Soto Setzke & Manuel Wiesche & Markus Böhm & Helmut Krcmar, 2020. "Digital platform ecosystems," Electronic Markets, Springer;IIM University of St. Gallen, vol. 30(1), pages 87-98, March.
    18. Van Dyck, Marc & Lüttgens, Dirk & Diener, Kathleen & Piller, Frank & Pollok, Patrick, 2024. "From product to platform: How incumbents' assumptions and choices shape their platform strategy," Research Policy, Elsevier, vol. 53(1).
    19. Fabian Schueler & Dimitri Petrik, 2022. "Objectives of platform research: A co-citation and systematic literature review analysis," Papers 2202.08822, arXiv.org.
    20. Knudsen, Eirik Sjåholm & Lien, Lasse B. & Timmermans, Bram & Belik, Ivan & Pandey, Sujit, 2021. "Stability in turbulent times? The effect of digitalization on the sustainability of competitive advantage," Journal of Business Research, Elsevier, vol. 128(C), pages 360-369.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525003063. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.