IDEAS home Printed from https://ideas.repec.org/a/sgh/annals/i41y2016p189-202.html
   My bibliography  Save this article

Econometric modeling of panel data using parallel computing with Apache Spark

Author

Listed:
  • Michał Bernardelli

    (Warsaw School of Economics)

Abstract

The aim of this article is to provide a method for determining the fixed effects estimators using MapReduce programming model implemented in Apache Spark. From many known algorithms two common approaches were exploited: the within transformation and least squares dummy variables method (LSDV). Efficiency of the computations was demonstrated by solving a specially crafted example for sample data. Based on theoretical analysis and computer experiments it can be stated that Apache Spark is an efficient tool for modeling panel data especially if it comes to Big Data.

Suggested Citation

  • Michał Bernardelli, 2016. "Econometric modeling of panel data using parallel computing with Apache Spark," Collegium of Economic Analysis Annals, Warsaw School of Economics, Collegium of Economic Analysis, issue 41, pages 189-202.
    • Handle: RePEc:sgh:annals:i:41:y:2016:p:189-202
    as

    Download full text from publisher

    File URL: http://rocznikikae.sgh.waw.pl/p/roczniki_kae_z41_12.pdf
    File Function: Full text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Arellano, Manuel, 2003. "Panel Data Econometrics," OUP Catalogue, Oxford University Press, number 9780199245291.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Michał Bernardelli, 2018. "The method of examining the properties of transition rules for bonus-malus systems using Apache Spark," Collegium of Economic Analysis Annals, Warsaw School of Economics, Collegium of Economic Analysis, issue 51, pages 95-108.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:hal:spmain:info:hdl:2441/dambferfb7dfprc9m052g20qh is not listed on IDEAS
    2. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja & Tristan Zajonc, 2011. "Do Value-Added Estimates Add Value? Accounting for Learning Dynamics," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 29-54, July.
    3. Carrión-Flores, Carmen E. & Innes, Robert, 2010. "Environmental innovation and environmental performance," Journal of Environmental Economics and Management, Elsevier, vol. 59(1), pages 27-42, January.
    4. Paul Raschky, 2007. "Estimating the effects of risk transfer mechanisms against floods in Europe and U.S.A.: A dynamic panel approach," Working Papers 2007-05, Faculty of Economics and Statistics, Universität Innsbruck.
    5. Giacomo De Giorgi & Michele Pellizzari & William Gui Woolston, 2012. "Class Size And Class Heterogeneity," Journal of the European Economic Association, European Economic Association, vol. 10(4), pages 795-830, August.
    6. Rode, Martin & Gwartney, James D., 2012. "Does democratization facilitate economic liberalization?," European Journal of Political Economy, Elsevier, vol. 28(4), pages 607-619.
    7. Anne Musson & Damien Rousselière, 2020. "Exploring the effect of crisis on cooperatives: a Bayesian performance analysis of French craftsmen cooperatives," Applied Economics, Taylor & Francis Journals, vol. 52(25), pages 2657-2678, May.
    8. Iván Fernández-Val & Martin Weidner, 2018. "Fixed Effects Estimation of Large-TPanel Data Models," Annual Review of Economics, Annual Reviews, vol. 10(1), pages 109-138, August.
    9. Xiaohong Chen & Andres Santos, 2018. "Overidentification in Regular Models," Econometrica, Econometric Society, vol. 86(5), pages 1771-1817, September.
    10. Johannes Blum & Klaus Gründler, 2020. "Political Stability and Economic Prosperity: Are Coups Bad for Growth?," CESifo Working Paper Series 8317, CESifo.
    11. Marktanner Marcus & Makdisi Samir, 2008. "Development against All Odds? The Case of Lebanon," Review of Middle East Economics and Finance, De Gruyter, vol. 4(3), pages 101-133, September.
    12. Gabriel Burdí­n & Andrés Dean, 2009. "Las decisiones de empleo y salarios de cooperativas de trabajo y empresas capitalistas : evidencia para Uruguay en base a datos de panel," Documentos de Trabajo (working papers) 09-02, Instituto de Economía - IECON.
    13. Maxime Fajeau, 2020. "The Adverse Effect of Finance on Growth," Working Papers hal-02549422, HAL.
    14. Hoderlein, Stefan & White, Halbert, 2012. "Nonparametric identification in nonseparable panel data models with generalized fixed effects," Journal of Econometrics, Elsevier, vol. 168(2), pages 300-314.
    15. García Cruz Gustavo Adolfo, 2008. "Informalidad regional en Colombia. Evidencia y Determinantes," Revista Desarrollo y Sociedad, Universidad de los Andes,Facultad de Economía, CEDE, February.
    16. Karlsson, Martin & Nilsson, Therese & Pichler, Stefan, 2014. "The impact of the 1918 Spanish flu epidemic on economic performance in Sweden," Journal of Health Economics, Elsevier, vol. 36(C), pages 1-19.
    17. Samargandi, Nahla & Fidrmuc, Jan & Ghosh, Sugata, 2015. "Is the Relationship Between Financial Development and Economic Growth Monotonic? Evidence from a Sample of Middle-Income Countries," World Development, Elsevier, vol. 68(C), pages 66-81.
    18. Julie L. Hotchkiss & Robert E. Moore, 2022. "Some Like it Hot: Assessing Longer-Term Labor Market Benefits from a High-Pressure Economy," International Journal of Central Banking, International Journal of Central Banking, vol. 18(2), pages 193-243, June.
    19. Troske, Kenneth R. & Voicu, Alexandru, 2010. "Joint estimation of sequential labor force participation and fertility decisions using Markov chain Monte Carlo techniques," Labour Economics, Elsevier, vol. 17(1), pages 150-169, January.
    20. Pablo Lavado & Gonzalo Rivera, 2016. "Identifying Treatment Effects with Data Combination and Unobserved Heterogeneity," Working Papers 79, Peruvian Economic Association.
    21. Christopher B. Goodman, 2015. "Local Government Fragmentation and the Local Public Sector," Public Finance Review, , vol. 43(1), pages 82-107, January.

    More about this item

    Keywords

    fixed effects estimator; panel data; Apache Spark; Big Data; MapReduce;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sgh:annals:i:41:y:2016:p:189-202. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michał Bernardelli (email available below). General contact details of provider: https://edirc.repec.org/data/sgwawpl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.