IDEAS home Printed from https://ideas.repec.org/a/taf/japsta/v42y2015i12p2554-2570.html
   My bibliography  Save this article

Dealing with the biased effects issue when handling huge datasets: the case of INVALSI data

Author

Listed:
  • E. Raffinetti
  • I. Romeo

Abstract

The increasing prevalence of huge datasets addresses the research to appropriate statistical methods for solving troubles caused by their complexity. On the one hand, several techniques are mentioned in the literature, especially for the time-consuming and variables reduction issues. On the other, less debate is devoted to the statistical inference issue. Indeed, a large number of involved statistical units may lead to wrongly consider as significant variables without any actual impact on the phenomenon under study. This paper suggests a suitable subsampling procedure for the reduction of the number of statistical units and provides a novel index for the assessment of the significance effects. The proposal is validated by comparing results obtained from the analysis on the original data to those obtained from the proposed subsampling approach. The illustrative application focuses on the educational dataset made available by the National Committee for the Evaluation of the Italian Education Systems (INVALSI). This dataset collects information about the student features and achievements in Maths within the lower secondary schools of the Lombardy region (Italy). Due to the hierarchical structure of the data, a multilevel model is implemented with the purpose of investigating the effects of both individual and school factors on student Maths score.

Suggested Citation

  • E. Raffinetti & I. Romeo, 2015. "Dealing with the biased effects issue when handling huge datasets: the case of INVALSI data," Journal of Applied Statistics, Taylor & Francis Journals, vol. 42(12), pages 2554-2570, December.
    • Handle: RePEc:taf:japsta:v:42:y:2015:i:12:p:2554-2570
    DOI: 10.1080/02664763.2015.1043867
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/02664763.2015.1043867
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/02664763.2015.1043867?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. S. K. Vines, 2000. "Simple principal components," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 49(4), pages 441-451.
    2. Hugh Chipman & Hong Gu, 2005. "Interpretable dimension reduction," Journal of Applied Statistics, Taylor & Francis Journals, vol. 32(9), pages 969-987.
    3. Kapetanios, George, 2010. "A Testing Procedure for Determining the Number of Factors in Approximate Factor Models With Large Datasets," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(3), pages 397-409.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paolo Giudici & Emanuela Raffinetti, 2021. "Cyber risk ordering with rank-based statistical models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 105(3), pages 469-484, September.
    2. Malavasi, Matteo & Peters, Gareth W. & Shevchenko, Pavel V. & Trück, Stefan & Jang, Jiwook & Sofronov, Georgy, 2022. "Cyber risk frequency, severity and insurance viability," Insurance: Mathematics and Economics, Elsevier, vol. 106(C), pages 90-114.
    3. Matteo Malavasi & Gareth W. Peters & Pavel V. Shevchenko & Stefan Truck & Jiwook Jang & Georgy Sofronov, 2021. "Cyber Risk Frequency, Severity and Insurance Viability," Papers 2111.03366, arXiv.org, revised Mar 2022.
    4. Daniel Doz & Mara Cotič & Darjo Felda, 2023. "Random Forest Regression in Predicting Students’ Achievements and Fuzzy Grades," Mathematics, MDPI, vol. 11(19), pages 1-19, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Trendafilov, Nickolay T. & Vines, Karen, 2009. "Simple and interpretable discrimination," Computational Statistics & Data Analysis, Elsevier, vol. 53(4), pages 979-989, February.
    2. T. F. Cox & D. S. Arnold, 2018. "Simple components," Journal of Applied Statistics, Taylor & Francis Journals, vol. 45(1), pages 83-99, January.
    3. Nickolay Trendafilov, 2014. "From simple structure to sparse components: a review," Computational Statistics, Springer, vol. 29(3), pages 431-454, June.
    4. GUO-FITOUSSI, Liang, 2013. "A Comparison of the Finite Sample Properties of Selection Rules of Factor Numbers in Large Datasets," MPRA Paper 50005, University Library of Munich, Germany.
    5. Claudio Morana, 2014. "Factor Vector Autoregressive Estimation of Heteroskedastic Persistent and Non Persistent Processes Subject to Structural Breaks," Working Papers 273, University of Milano-Bicocca, Department of Economics, revised May 2014.
    6. Matteo Barigozzi & Antonio M. Conti & Matteo Luciani, 2014. "Do Euro Area Countries Respond Asymmetrically to the Common Monetary Policy?," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 76(5), pages 693-714, October.
    7. Kim, Hyun Hak & Swanson, Norman R., 2018. "Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods," International Journal of Forecasting, Elsevier, vol. 34(2), pages 339-354.
    8. Alain-Philippe Fortin & Patrick Gagliardini & O. Scaillet, 2022. "Eigenvalue tests for the number of latent factors in short panels," Swiss Finance Institute Research Paper Series 22-81, Swiss Finance Institute.
    9. Wei, Jie & Chen, Hui, 2020. "Determining the number of factors in approximate factor models by twice K-fold cross validation," Economics Letters, Elsevier, vol. 191(C).
    10. Yongfu Huang & Muhammad G. Quibria, 2015. "The global partnership for sustainable development," Natural Resources Forum, Blackwell Publishing, vol. 0(3-4), pages 157-174, August.
    11. Gagliardini, Patrick & Ossola, Elisa & Scaillet, Olivier, 2019. "A diagnostic criterion for approximate factor structure," Journal of Econometrics, Elsevier, vol. 212(2), pages 503-521.
    12. Xu Cheng & Zhipeng Liao & Frank Schorfheide, 2016. "Shrinkage Estimation of High-Dimensional Factor Models with Structural Instabilities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 83(4), pages 1511-1543.
    13. Yunus Emre Ergemen & Carlos Vladimir Rodríguez-Caballero, 2016. "A Dynamic Multi-Level Factor Model with Long-Range Dependence," CREATES Research Papers 2016-23, Department of Economics and Business Economics, Aarhus University.
    14. Gu, Shihao & Kelly, Bryan & Xiu, Dacheng, 2021. "Autoencoder asset pricing models," Journal of Econometrics, Elsevier, vol. 222(1), pages 429-450.
    15. Tae-Hwy Lee & Ekaterina Seregina, 2020. "Optimal Portfolio Using Factor Graphical Lasso," Working Papers 202025, University of California at Riverside, Department of Economics.
    16. Claudio Barbieri & Mattia Guerini & Mauro Napoletano, 2021. "The anatomy of government bond yields synchronization in the Eurozone," SciencePo Working papers Main hal-03373853, HAL.
    17. Guowei Cui & Milda NorkutÄ— & Vasilis Sarafidis & Takashi Yamagata, 2022. "Two-stage instrumental variable estimation of linear panel data models with interactive effects [Eigenvalue ratio test for the number of factors]," The Econometrics Journal, Royal Economic Society, vol. 25(2), pages 340-361.
    18. Cynthia Fan Yang, 2021. "Common factors and spatial dependence: an application to US house prices," Econometric Reviews, Taylor & Francis Journals, vol. 40(1), pages 14-50, January.
    19. Andres Sagner, 2020. "High Dimensional Quantile Factor Analysis," Working Papers Central Bank of Chile 886, Central Bank of Chile.
    20. Ergemen, Yunus Emre & Rodríguez-Caballero, C. Vladimir, 2023. "Estimation of a dynamic multi-level factor model with possible long-range dependence," International Journal of Forecasting, Elsevier, vol. 39(1), pages 405-430.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:japsta:v:42:y:2015:i:12:p:2554-2570. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/CJAS20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.