M2F-Net: A Deep Learning-Based Multimodal Classification with High-Throughput Phenotyping for Identification of Overabundance of Fertilizers

Amaranth, a pseudocereal crop which is rich in nutrients and climate resistant, can provide an opportunity to increase food security and nutritional content for the growing population. Farmers rely mainly on synthetic fertilizers to improve the quality and yield of the crop; however, this overuse harms the ecosystem. Understanding the mechanism causing this environmental deterioration is crucial for crop production and ecological sustainability. In recent years, high-throughput phenotyping using Artificial Intelligence (AI) has been thriving and can provide an effective solution for the identification of fertilizer overuse. Influenced by the strength of deep learning paradigms and IoT sensors, a novel multimodal fusion network (M2F-Net) is proposed for high-throughput phenotyping to diagnose overabundance of fertilizers. In this paper, we developed and analyzed three strategies that fuse agrometeorological and image data by assessing fusion at various stages. Initially two unimodal baseline networks were trained: Multi-Layer Perceptron (MLP) on agrometeorological data and a pre-trained Convolutional Neural Network (CNN) model DenseNet-121 on image data. With these baselines, the multimodal fusion network is developed, capable of adeptly learning from image and non-image data and the model’s performance is evaluated in terms of accuracy and Area Under Curve (AUC). Moreover, the fusion approaches that are considered outperformed the unimodal networks remarkably with 91% accuracy. From the experimental result, it is proven that incorporating agrometeorological information and images can substantially boost the classification performance for the overabundance of fertilizer.