IDEAS home Printed from https://ideas.repec.org/a/gam/jagris/v13y2023i5p1015-d1140420.html
   My bibliography  Save this article

An Approach Based on Web Scraping and Denoising Encoders to Curate Food Security Datasets

Author

Listed:
  • Fabián Santos

    (Centro de Investigación Para el Territorio y el Hábitat Sostenible (CITEHS), Universidad Indoamérica, Quito 170301, Ecuador)

  • Nicole Acosta

    (Research Unit Sustainability and Climate Risks, Universität Hamburg, 20144 Hamburg, Germany)

Abstract

Ensuring food security requires the publication of data in a timely manner, but often this information is not properly documented and evaluated. Therefore, the combination of databases from multiple sources is a common practice to curate the data and corroborate the results; however, this also results in incomplete cases. These tasks are often labor-intensive since they require a case-wise review to obtain the requested and completed information. To address these problems, an approach based on Selenium web-scraping software and the multiple imputation denoising autoencoders (MIDAS) algorithm is presented for a case study in Ecuador. The objective was to produce a multidimensional database, free of data gaps, with 72 species of food crops based on the data from 3 different open data web databases. This methodology resulted in an analysis-ready dataset with 43 parameters describing plant traits, nutritional composition, and planted areas of food crops, whose imputed data obtained an R-square of 0.84 for a control numerical parameter selected for validation. This enriched dataset was later clustered with K-means to report unprecedented insights into food crops cultivated in Ecuador. The methodology is useful for users who need to collect and curate data from different sources in a semi-automatic fashion.

Suggested Citation

  • Fabián Santos & Nicole Acosta, 2023. "An Approach Based on Web Scraping and Denoising Encoders to Curate Food Security Datasets," Agriculture, MDPI, vol. 13(5), pages 1-19, May.
    • Handle: RePEc:gam:jagris:v:13:y:2023:i:5:p:1015-:d:1140420
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2077-0472/13/5/1015/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2077-0472/13/5/1015/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ania Cravero & Sebastián Pardo & Patricio Galeas & Julio López Fenner & Mónica Caniupán, 2022. "Data Type and Data Sources for Agricultural Big Data and Machine Learning," Sustainability, MDPI, vol. 14(23), pages 1-37, December.
    2. Herforth, Anna & Venkat, Aishwarya & Bai, Yan & Costlow, Leah & Holleman, Cindy & Masters, William A., 2022. "Methods and options to monitor the cost and affordability of a healthy diet globally Background paper for The State of Food Security and Nutrition in the World 2022," ESA Working Papers 324075, Food and Agriculture Organization of the United Nations, Agricultural Development Economics Division (ESA).
    3. Robert Thorndike, 1953. "Who belongs in the family?," Psychometrika, Springer;The Psychometric Society, vol. 18(4), pages 267-276, December.
    4. Lumley, Thomas, 2004. "Analysis of Complex Survey Samples," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i08).
    5. Wickham, Hadley, 2007. "Reshaping Data with the reshape Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 21(i12).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang, Ling, 2016. "Is Bank Supervision Effective? Evidence from the Allowance for Loan and Lease Losses," MPRA Paper 75761, University Library of Munich, Germany.
    2. Koecklin, Manuel Tong & Longoria, Genaro & Fitiwi, Desta Z. & DeCarolis, Joseph F. & Curtis, John, 2021. "Public acceptance of renewable electricity generation and transmission network developments: Insights from Ireland," Energy Policy, Elsevier, vol. 151(C).
    3. Maciej Berk{e}sewicz & Herman Cherniaiev & Robert Pater, 2021. "Estimating the number of entities with vacancies using administrative and online data," Papers 2106.03263, arXiv.org.
    4. Augustinus, Benno A. & Blum, Moshe & Citterio, Sandra & Gentili, Rodolfo & Helman, David & Nestel, David & Schaffner, Urs & Müller-Schärer, Heinz & Lensky, Itamar M., 2022. "Ground-truthing predictions of a demographic model driven by land surface temperatures with a weed biocontrol cage experiment," Ecological Modelling, Elsevier, vol. 466(C).
    5. Becken, Susanne & Stantic, Bela & Chen, Jinyan & Connolly, Rod M., 2022. "Twitter conversations reveal issue salience of aviation in the broader context of climate change," Journal of Air Transport Management, Elsevier, vol. 98(C).
    6. Rockstuhl, Sebastian & Wenninger, Simon & Wiethe, Christian & Ahlrichs, Jakob, 2022. "The influence of risk perception on energy efficiency investments: Evidence from a German survey," Energy Policy, Elsevier, vol. 167(C).
    7. Tong Koecklin, Manuel & Fitiwi, Desta & de Carolis, Joseph F. & Curtis, John, 2020. "Renewable electricity generation and transmission network developments in light of public opposition: Insights from Ireland," Papers WP653, Economic and Social Research Institute (ESRI).
    8. Julio Cesar Alonso Cifuentes & Jaime Andres Carabali, 2019. "Breve Tuturial para visualizar y Calcular Métricas de Redes (grafos) en R (para Económisas)," Icesi Economics Lecture Notes 18170, Universidad Icesi.
    9. J. Michael Brick & Michael E. Jones, 2008. "Propensity to respond and nonresponse bias," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(1), pages 51-73.
    10. Archana R. Panhalkar & Dharmpal D. Doye, 2020. "An approach of improving decision tree classifier using condensed informative data," DECISION: Official Journal of the Indian Institute of Management Calcutta, Springer;Indian Institute of Management Calcutta, vol. 47(4), pages 431-445, December.
    11. Jacques Muthusi & Samuel Mwalili & Peter Young, 2019. "%svy_logistic_regression: A generic SAS macro for simple and multiple logistic regression and creating quality publication-ready tables using survey or non-survey data," PLOS ONE, Public Library of Science, vol. 14(9), pages 1-14, September.
    12. Michele Cincera, 2005. "Firms' productivity growth and R&D spillovers: An analysis of alternative technological proximity measures," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 14(8), pages 657-682.
    13. Horstmann, Felix, 2017. "Measuring the shopper's attitude toward the point of sale display: Scale development and validation," Journal of Retailing and Consumer Services, Elsevier, vol. 36(C), pages 112-123.
    14. Jonathan Wakefield & Taylor Okonek & Jon Pedersen, 2020. "Small Area Estimation for Disease Prevalence Mapping," International Statistical Review, International Statistical Institute, vol. 88(2), pages 398-418, August.
    15. Elizaveta Zinovyeva & Raphael C. G. Reule & Wolfgang Karl Hardle, 2021. "Understanding Smart Contracts: Hype or Hope?," Papers 2103.08447, arXiv.org.
    16. Alex Fenton, 2013. "Small-area measures of income poverty," CASE Papers case173, Centre for Analysis of Social Exclusion, LSE.
    17. repec:cep:sticas:/173 is not listed on IDEAS
    18. Mei‐Chih Meg Tseng & Yi‐Ping Lin & Fu‐Chang Hu & Tsun‐Jen Cheng, 2013. "Risks Perception of Electromagnetic Fields in Taiwan: The Influence of Psychopathology and the Degree of Sensitivity to Electromagnetic Fields," Risk Analysis, John Wiley & Sons, vol. 33(11), pages 2002-2012, November.
    19. Raphael Nishimura & James Wagner & Michael Elliott, 2016. "Alternative Indicators for the Risk of Non-response Bias: A Simulation Study," International Statistical Review, International Statistical Institute, vol. 84(1), pages 43-62, April.
    20. Miller, Christine M.F. & Waterhouse, Hannah & Harter, Thomas & Fadel, James G. & Meyer, Deanne, 2020. "Quantifying the uncertainty in nitrogen application and groundwater nitrate leaching in manure based cropping systems," Agricultural Systems, Elsevier, vol. 184(C).
    21. Sarlas, Georgios & Páez, Antonio & Axhausen, Kay W., 2020. "Betweenness-accessibility: Estimating impacts of accessibility on networks," Journal of Transport Geography, Elsevier, vol. 84(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jagris:v:13:y:2023:i:5:p:1015-:d:1140420. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.