IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v17y2025i8p3727-d1638701.html
   My bibliography  Save this article

Design and Implementation of a Scalable Data Warehouse for Agricultural Big Data

Author

Listed:
  • Asterios Theofilou

    (Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece)

  • Stefanos A. Nastis

    (Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece)

  • Michail Tsagris

    (Department of Economics, University of Crete, 74100 Rethymno, Greece)

  • Santiago Rodriguez-Perez

    (Biotechnology Applications, IDENER, Early Ovington 24 Nave 8-9, 41300 Seville, Spain)

  • Konstadinos Mattas

    (Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece)

Abstract

The rapid growth of agricultural data necessitates the development of storage systems that are scalable and efficient in storing, retrieving and analyzing very large datasets. The traditional relational database management systems (RDBMSs) struggle to keep up with large-scale analytical queries due to the volume and complexity inherent in those data. This study presents the design and implementation of a scalable data warehouse (DWH) system for agricultural big data. The proposed solution efficiently integrates data and optimizes data ingestion, transformation, and query performance, leveraging a distributed architecture based on HDFS, Apache Hive, and Apache Spark, deployed on dockerized Ubuntu Linux environments. This paper highlights the reasons why a DWH is irreplaceable for big data processing, without disputing the strengths of traditional databases in transactional use cases. By detailing the architectural choices and implementation strategy, this study provides a practical framework for deploying robust DWH solutions that are useful in supporting agricultural research, market predictions and policy decision-making.

Suggested Citation

  • Asterios Theofilou & Stefanos A. Nastis & Michail Tsagris & Santiago Rodriguez-Perez & Konstadinos Mattas, 2025. "Design and Implementation of a Scalable Data Warehouse for Agricultural Big Data," Sustainability, MDPI, vol. 17(8), pages 1-19, April.
    • Handle: RePEc:gam:jsusta:v:17:y:2025:i:8:p:3727-:d:1638701
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/17/8/3727/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2071-1050/17/8/3727/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mario San Emeterio de la Parte & Sara Lana Serrano & Marta Muriel Elduayen & José-Fernán Martínez-Ortega, 2023. "Spatio-Temporal Semantic Data Model for Precision Agriculture IoT Networks," Agriculture, MDPI, vol. 13(2), pages 1-28, February.
    2. Osinga, Sjoukje A. & Paudel, Dilli & Mouzakitis, Spiros A. & Athanasiadis, Ioannis N., 2022. "Big data in agriculture: Between opportunity and solution," Agricultural Systems, Elsevier, vol. 195(C).
    3. Shengbin Hao & Haili Zhang & Michael Song, 2019. "Big Data, Big Data Analytics Capability, and Sustainable Innovation Performance," Sustainability, MDPI, vol. 11(24), pages 1-15, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeroen Ooge & Katrien Verbert, 2022. "Visually Explaining Uncertain Price Predictions in Agrifood: A User-Centred Case-Study," Agriculture, MDPI, vol. 12(7), pages 1-25, July.
    2. Haili Zhang & Yufan Wang & Michael Song, 2019. "Does Competitive Intensity Moderate the Relationships between Sustainable Capabilities and Sustainable Organizational Performance in New Ventures?," Sustainability, MDPI, vol. 12(1), pages 1-18, December.
    3. Mihai BOGDAN & Anca BORZA, 2020. "Big Data Analytics And Firm Performance: A Text Mining Approach," Proceedings of the INTERNATIONAL MANAGEMENT CONFERENCE, Faculty of Management, Academy of Economic Studies, Bucharest, Romania, vol. 14(1), pages 549-560, November.
    4. Haili Zhang & Michael Song & Huanhuan He, 2020. "Achieving the Success of Sustainability Development Projects through Big Data Analytics and Artificial Intelligence Capability," Sustainability, MDPI, vol. 12(3), pages 1-23, January.
    5. Qidi Dong & Jun Cai & Shuo Chen & Pengman He & Xuli Chen, 2022. "Spatiotemporal Analysis of Urban Green Spatial Vitality and the Corresponding Influencing Factors: A Case Study of Chengdu, China," Land, MDPI, vol. 11(10), pages 1-17, October.
    6. Zhengang Zhang & Yu Shang & Linyuan Cheng & Antao Hu, 2022. "Big Data Capability and Sustainable Competitive Advantage: The Mediating Role of Ambidextrous Innovation Strategy," Sustainability, MDPI, vol. 14(14), pages 1-17, July.
    7. Hua Zhang & Shaofeng Yuan, 2023. "How and When Does Big Data Analytics Capability Boost Innovation Performance?," Sustainability, MDPI, vol. 15(5), pages 1-19, February.
    8. Xiaoli Wang & Ying Gu & Mahmood Ahmad & Chaokai Xue, 2022. "The Impact of Digital Capability on Manufacturing Company Performance," Sustainability, MDPI, vol. 14(10), pages 1-24, May.
    9. Yifei Yang & Dapeng Lian & Yanan Zhang & Dongxuan Wang & Jianzhong Wang, 2024. "Towards Sustainable Agricultural Development: Integrating Small-Scale Farmers in China Through Agricultural Social Services," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(4), pages 16575-16616, December.
    10. Miloš Arsić & Zoran Jovanović & Radoljub Tomić & Nena Tomović & Siniša Arsić & Ištvan Bodolo, 2020. "Impact of Logistics Capacity on Economic Sustainability of SMEs," Sustainability, MDPI, vol. 12(5), pages 1-30, March.
    11. Huynh, Minh-Tay & Nippa, Michael & Aichner, Thomas, 2023. "Big data analytics capabilities: Patchwork or progress? A systematic review of the status quo and implications for future research," Technological Forecasting and Social Change, Elsevier, vol. 197(C).
    12. Irina Bogdana Pugna & Dana Maria Boldeanu & Mirela Gheorghe & Gabriel Cozgarea & Adrian Nicolae Cozgarea, 2022. "Management Perspectives towards the Data-Driven Organization in the Energy Sector," Energies, MDPI, vol. 15(16), pages 1-20, August.
    13. Zhen Wang & Chunhui Yuan & Xiaolong Li, 2024. "Unleashing the power of big data for platform firms: A configuration analysis," Managerial and Decision Economics, John Wiley & Sons, Ltd., vol. 45(1), pages 300-314, January.
    14. Zhang, Chen & Di, Liping & Lin, Li & Li, Hui & Guo, Liying & Yang, Zhengwei & Yu, Eugene G. & Di, Yahui & Yang, Anna, 2022. "Towards automation of in-season crop type mapping using spatiotemporal crop information and remote sensing data," Agricultural Systems, Elsevier, vol. 201(C).
    15. JAHAN Sakila Akter & SAZU Mesbaul Haque, 2022. "Innovation Management: Is Big Data Necessarily Better Data?," Management of Sustainable Development, Lucian Blaga University of Sibiu, Faculty of Economic Sciences, vol. 14(2), pages 27-33, December.
    16. Philipp Korherr & Dominik Kanbach, 2023. "Human-related capabilities in big data analytics: a taxonomy of human factors with impact on firm performance," Review of Managerial Science, Springer, vol. 17(6), pages 1943-1970, August.
    17. Meng Zhang & Yong Qi, 2023. "Vertical Network Relationships, Technological Capabilities, and Innovation Performance: The Moderating Role of Strategic Flexibility," Sustainability, MDPI, vol. 15(14), pages 1-15, July.
    18. Guixiang Cao & Xintong Fang & Ying Chen & Jinghuai She, 2023. "Regional Big Data Application Capability and Firm Green Technology Innovation," Sustainability, MDPI, vol. 15(17), pages 1-29, August.
    19. S. M. F. D. Syed Mustapha, 2022. "The UAE Employees’ Perceptions towards Factors for Sustaining Big Data Implementation and Continuous Impact on Their Organization’s Performance," Sustainability, MDPI, vol. 14(22), pages 1-19, November.
    20. Gniewko Niedbała & Sebastian Kujawa, 2023. "Digital Innovations in Agriculture," Agriculture, MDPI, vol. 13(9), pages 1-10, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:17:y:2025:i:8:p:3727-:d:1638701. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.