Unlocking plant health survey data: An approach to quantify the sensitivity and specificity of visual inspections

Invasive plant pests and pathogens cause substantial environmental and economic damage. Visual inspection remains a central tenet of plant health surveys, but its sensitivity (probability of correctly identifying the presence of a pest) and specificity (probability of correctly identifying the absence of a pest) are not routinely quantified. As knowing sensitivity and specificity of visual inspection is critical for effective contingency planning and outbreak management, we address this deficiency using empirical data and statistical analyses. Twenty-three citizen scientist surveyors assessed up to 175 labelled oak trees for three symptoms of acute oak decline. The same trees were also assessed by an expert who has monitored these individual trees annually for over a decade. The sensitivity and specificity of surveyors was calculated using the expert data as the ‘gold-standard’ (i.e., assuming perfect sensitivity and specificity). The utility of an approach using Bayesian modelling to estimate the sensitivity and specificity of visual inspection in the absence of a rarely available ‘gold-standard’ dataset was then examined with simulated plant health survey datasets. There was large variation in sensitivity and specificity between surveyors and between different symptoms, although the sensitivity of detecting a symptom was positively related to the frequency of the symptom on a tree. By leveraging surveyor observations of two symptoms from a minimum of 80 trees on two sites, with reliable prior knowledge of sites with a higher (~0.6) and lower (~0.3) true disease prevalence we show that sensitivity and specificity can be estimated without ‘gold-standard’ data using Bayesian modelling. We highlight that sensitivity and specificity will depend on the symptoms of a pest or disease, the individual surveyor, and the survey protocol. This has consequences for how surveys are designed to detect and monitor outbreaks, as well as the interpretation of survey data that is used to inform outbreak management.Author summary: The increasing occurrence of emerging plant pests and diseases is affecting both agricultural and natural ecosystems. Effective management and control of such pests and diseases is much easier when they are detected early. Currently, visual surveys underpin plant health surveillance, but basic metrics of the reliability of visual detection such as the sensitivity (probability of correctly identifying a positive) and specificity (probability of correctly identifying a negative) are not routinely quantified. In this study, we first quantify the sensitivity and specificity of 23 trained citizen scientist surveyors at detecting three symptoms of acute oak decline, by comparing their symptom classifications against a dataset from an expert who has conducted long-term monitoring of these individual trees. We demonstrate how individuals vary greatly in their ability to detect symptoms, and how different symptoms are associated with different detection error. Secondly, based on this dataset we outline an approach developed for scenarios realistic in plant health which utilises Bayesian modelling to estimate the sensitivity and specificity in the absence of a rarely available ‘gold-standard’ (i.e., assuming perfect sensitivity and specificity) expert dataset. In summary, our results highlight variation in the reliability of visual detection, and we provide an approach to calculate this and facilitate optimisation of risk-based surveillance strategies in plant health.