IDEAS home Printed from https://ideas.repec.org/p/zbw/itse11/52173.html
   My bibliography  Save this paper

Web crawler research methodology

Author

Listed:
  • Nemeslaki, András
  • Pocsarovszky, Károly

Abstract

In economic and social sciences it is crucial to test theoretical models against reliable and big enough databases. The general research challenge is to build up a well-structured database that suits well to the given research question and that is cost efficient at the same time. In this paper we focus on crawler programs that proved to be an effective tool of data base building in very different problem settings. First we explain how crawler programs work and illustrate a complex research process mapping business relationships using social media information sources. In this case we illustrate how search robots can be used to collect data for mapping complex network relationship to characterize business relationships in a well defined environment. After that extend the case and present a framework of three structurally different research models where crawler programs can be applied successfully: exploration, classification and time series analysis. In the case of exploration we present findings about the Hungarian web agency industry when no previous statistical data was available about their operations. For classification we show how the top visited Hungarian web domains can be divided into predefined categories of e-business models. In the third research we used a crawler to gather the values of concrete pre-defined records containing ticket prices of low cost airlines from one single site. Based on the experiences we highlight some conceptual conclusions and opportunities of crawler based research in e-business.

Suggested Citation

  • Nemeslaki, András & Pocsarovszky, Károly, 2011. "Web crawler research methodology," 22nd European Regional ITS Conference, Budapest 2011: Innovative ICT Applications - Emerging Regulatory, Economic and Policy Issues 52173, International Telecommunications Society (ITS).
  • Handle: RePEc:zbw:itse11:52173
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/52173/1/67255853X.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Steffen Hoernig, 2006. "Should uniform pricing constraints be imposed on entrants?," Journal of Regulatory Economics, Springer, vol. 30(2), pages 199-216, August.
    2. Briglauer Wolfgang & Götz Georg & Schwarz Anton, 2011. "Margin Squeeze in Fixed-Network Telephony Markets -- Competitive or Anticompetitive?," Review of Network Economics, De Gruyter, vol. 10(4), pages 1-21, December.
    3. Varian, Hal R, 1980. "A Model of Sales," American Economic Review, American Economic Association, vol. 70(4), pages 651-659, September.
    4. Bender, Christian M. & Götz, Georg, 2010. "Coexistence of service- and facility-based competition: The relevance of access prices for "make-or-buy"-decisions," 21st European Regional ITS Conference, Copenhagen 2010: Telecommunications at new crossroads - Changing value configurations, user roles, and regulation 3, International Telecommunications Society (ITS).
    5. Sangwon Lee & Justin S. Brown, 2008. "The Diffusion of Fixed Broadband: An Empirical Analysis," Working Papers 08-19, NET Institute, revised Sep 2008.
    6. Valletti, Tommaso M & Hoernig, Steffen & Barros, Pedro P, 2002. "Universal Service and Entry: The Role of Uniform Pricing and Coverage Constraints," Journal of Regulatory Economics, Springer, vol. 21(2), pages 169-190, March.
    7. Faulhaber, Gerald R & Hogendorn, Christiaan, 2000. "The Market Structure of Broadband Telecommunications," Journal of Industrial Economics, Wiley Blackwell, vol. 48(3), pages 305-329, September.
    8. Distaso, Walter & Lupi, Paolo & Manenti, Fabio M., 2006. "Platform competition and broadband uptake: Theory and empirical evidence from the European union," Information Economics and Policy, Elsevier, vol. 18(1), pages 87-106, March.
    9. Foros, Oystein & Kind, Hans Jarle, 2003. "The Broadband Access Market: Competition, Uniform Pricing and Geographical Coverage," Journal of Regulatory Economics, Springer, vol. 23(3), pages 215-235, May.
    10. Bouckaert, Jan & van Dijk, Theon & Verboven, Frank, 2010. "Access regulation, competition, and broadband penetration: An international study," Telecommunications Policy, Elsevier, pages 661-671.
    11. Mario Denni & Harald Gruber, 2006. "The diffusion of broadband telecommunications: the role of competition," Departmental Working Papers of Economics - University 'Roma Tre' 0060, Department of Economics - University Roma Tre.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    e-business research; web search; web crawler; Hungarian web; social network analyis;

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs
    • O3 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • L86 - Industrial Organization - - Industry Studies: Services - - - Information and Internet Services; Computer Software

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:itse11:52173. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (ZBW - German National Library of Economics). General contact details of provider: http://www.itseurope.org/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.