IDEAS home Printed from https://ideas.repec.org/a/gam/jpubli/v11y2023i4p49-d1279756.html
   My bibliography  Save this article

Migrating 120,000 Legacy Publications from Several Systems into a Current Research Information System Using Advanced Data Wrangling Techniques

Author

Listed:
  • Yrjö Lappalainen

    (Library and Learning Commons, Zayed University, Dubai P.O. Box 19282, United Arab Emirates)

  • Matti Lassila

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

  • Tanja Heikkilä

    (Finnish Geospatial Research Institute (FGI), National Land Survey of Finland (NLS), 02150 Espoo, Finland)

  • Jani Nieminen

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

  • Tapani Lehtilä

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

Abstract

This article describes a complex CRIS (current research information system) implementation project involving the migration of around 120,000 legacy publication records from three different systems. The project, undertaken by Tampere University, encountered several challenges in data diversity, data quality, and resource allocation. To handle the extensive and heterogenous dataset, innovative approaches such as machine learning techniques and various data wrangling tools were used to process data, correct errors, and merge information from different sources. Despite significant delays and unforeseen obstacles, the project was ultimately successful in achieving its goals. The project served as a valuable learning experience, highlighting the importance of data quality and standardized practices, and the need for dedicated resources in handling complex data migration projects in research organizations. This study stands out for its comprehensive documentation of the data wrangling and migration process, which has been less explored in the context of CRIS literature.

Suggested Citation

  • Yrjö Lappalainen & Matti Lassila & Tanja Heikkilä & Jani Nieminen & Tapani Lehtilä, 2023. "Migrating 120,000 Legacy Publications from Several Systems into a Current Research Information System Using Advanced Data Wrangling Techniques," Publications, MDPI, vol. 11(4), pages 1-16, November.
    • Handle: RePEc:gam:jpubli:v:11:y:2023:i:4:p:49-:d:1279756
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2304-6775/11/4/49/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2304-6775/11/4/49/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Helen Shen, 2014. "Interactive notebooks: Sharing the code," Nature, Nature, vol. 515(7525), pages 151-152, November.
    2. Kuhn, Max, 2008. "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 28(i05).
    3. Otmane Azeroual & Joachim Schöpfel, 2019. "Quality Issues of CRIS Data: An Exploratory Investigation with Universities from Twelve Countries," Publications, MDPI, vol. 7(1), pages 1-18, February.
    4. Helen Shen, 2014. "Interactive notebooks: Sharing the code," Nature, Nature, vol. 515(7525), pages 152-152, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ian L Morgan & Omar A Saleh, 2021. "Tweezepy: A Python package for calibrating forces in single-molecule video-tracking experiments," PLOS ONE, Public Library of Science, vol. 16(12), pages 1-18, December.
    2. Prabal Das & D. A. Sachindra & Kironmala Chanda, 2022. "Machine Learning-Based Rainfall Forecasting with Multiple Non-Linear Feature Selection Algorithms," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(15), pages 6043-6071, December.
    3. Paulo Infante & Gonçalo Jacinto & Anabela Afonso & Leonor Rego & Pedro Nogueira & Marcelo Silva & Vitor Nogueira & José Saias & Paulo Quaresma & Daniel Santos & Patrícia Góis & Paulo Rebelo Manuel, 2023. "Factors That Influence the Type of Road Traffic Accidents: A Case Study in a District of Portugal," Sustainability, MDPI, vol. 15(3), pages 1-16, January.
    4. Ephrem Habyarimana & Faheem S Baloch, 2021. "Machine learning models based on remote and proximal sensing as potential methods for in-season biomass yields prediction in commercial sorghum fields," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-23, March.
    5. Alexander Wettstein & Gabriel Jenni & Ida Schneider & Fabienne Kühne & Martin grosse Holtforth & Roberto La Marca, 2023. "Predictors of Psychological Strain and Allostatic Load in Teachers: Examining the Long-Term Effects of Biopsychosocial Risk and Protective Factors Using a LASSO Regression Approach," IJERPH, MDPI, vol. 20(10), pages 1-20, May.
    6. Tang, Kayu & Parsons, David J. & Jude, Simon, 2019. "Comparison of automatic and guided learning for Bayesian networks to analyse pipe failures in the water distribution system," Reliability Engineering and System Safety, Elsevier, vol. 186(C), pages 24-36.
    7. Daifeng Xiang & Gangsheng Wang & Jing Tian & Wanyu Li, 2023. "Global patterns and edaphic-climatic controls of soil carbon decomposition kinetics predicted from incubation experiments," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    8. Tranos, Emmanouil & Incera, Andre Carrascal & Willis, George, 2022. "Using the web to predict regional trade flows: data extraction, modelling, and validation," OSF Preprints 9bu5z, Center for Open Science.
    9. Štefan Lyócsa & Petra Vašaničová & Branka Hadji Misheva & Marko Dávid Vateha, 2022. "Default or profit scoring credit systems? Evidence from European and US peer-to-peer lending markets," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-21, December.
    10. Joachim Schöpfel & Stéphane Chaudiron & Bernard Jacquemin & Eric Kergosien & Hélène Prost & Florence Thiault, 2023. "The Transformation of the Green Road to Open Access," Publications, MDPI, vol. 11(2), pages 1-12, May.
    11. Marcos Rodrigues & Fermín Alcasena & Pere Gelabert & Cristina Vega‐García, 2020. "Geospatial Modeling of Containment Probability for Escaped Wildfires in a Mediterranean Region," Risk Analysis, John Wiley & Sons, vol. 40(9), pages 1762-1779, September.
    12. Giovanny Pillajo-Quijia & Blanca Arenas-Ramírez & Camino González-Fernández & Francisco Aparicio-Izquierdo, 2020. "Influential Factors on Injury Severity for Drivers of Light Trucks and Vans with Machine Learning Methods," Sustainability, MDPI, vol. 12(4), pages 1-28, February.
    13. Zander S. Venter & Adam Sadilek & Charlotte Stanton & David N. Barton & Kristin Aunan & Sourangsu Chowdhury & Aaron Schneider & Stefano Maria Iacus, 2021. "Mobility in Blue-Green Spaces Does Not Predict COVID-19 Transmission: A Global Analysis," IJERPH, MDPI, vol. 18(23), pages 1-12, November.
    14. G. Brooke Anderson & Keith W. Oleson & Bryan Jones & Roger D. Peng, 2018. "Classifying heatwaves: developing health-based models to predict high-mortality versus moderate United States heatwaves," Climatic Change, Springer, vol. 146(3), pages 439-453, February.
    15. Van Belle, Jente & Guns, Tias & Verbeke, Wouter, 2021. "Using shared sell-through data to forecast wholesaler demand in multi-echelon supply chains," European Journal of Operational Research, Elsevier, vol. 288(2), pages 466-479.
    16. Jun Wang & Jinyong Huang & Yunlong Hu & Qianwen Guo & Shasha Zhang & Jinglin Tian & Yanqin Niu & Ling Ji & Yuzhong Xu & Peijun Tang & Yaqin He & Yuna Wang & Shuya Zhang & Hao Yang & Kang Kang & Xinchu, 2024. "Terminal modifications independent cell-free RNA sequencing enables sensitive early cancer detection and classification," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    17. Ali Al-Ramini & Mohammad A Takallou & Daniel P Piatkowski & Fadi Alsaleem, 2022. "Quantifying changes in bicycle volumes using crowdsourced data," Environment and Planning B, , vol. 49(6), pages 1612-1630, July.
    18. Yagli, Gokhan Mert & Yang, Dazhi & Srinivasan, Dipti, 2019. "Automatic hourly solar forecasting using machine learning models," Renewable and Sustainable Energy Reviews, Elsevier, vol. 105(C), pages 487-498.
    19. Faisal Alsayegh & Moh A Alkhamis & Fatima Ali & Sreeja Attur & Nicholas M Fountain-Jones & Mohammad Zubaid, 2022. "Anemia or other comorbidities? using machine learning to reveal deeper insights into the drivers of acute coronary syndromes in hospital admitted patients," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-15, January.
    20. Breda, Thomas & Grenet, Julien & Monnet, Marion & Van Effenterre, Clémentine, 2020. "Do Female Role Models Reduce the Gender Gap in Science? Evidence from French High Schools," IZA Discussion Papers 13163, Institute of Labor Economics (IZA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jpubli:v:11:y:2023:i:4:p:49-:d:1279756. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.