Beyond big data – new techniques for forecasting elections using stochastic models with self-organisation and memory

This paper introduces an innovative social process model addressing population-wide measures of voter preferences that was tested on data from the 2016 US presidential election. Population-wide, "macroscopic" parameters are needed when privacy, ethics or regulatory constraints block "big data" techniques (e.g., in political contexts to counter "micro-targeting"). Confidence will be eroded if existing trend models and other macroscopic approaches frequently fail to predict outcomes, however campaign data reveal mathematical features that suggest a different possible approach. Given that the populations modelled exhibit self-organisation and memory when transmitting viewpoints, our model is based on mathematical representations of such processes. Its validation indicates the applicability and potential generalisability of this theoretical approach. In order to design a stochastic dynamics model of changing voter preferences, we evaluated probability models for transitions between possible system states (magnitudes of voter preferences), formulated the boundary task for probability density functions and derived a second-order non-linear differential equation incorporating self-organisation and memory. We find consistent dependencies between influences on the system and its reaction, and it is congruent with empirical data. The ability to use researchable global parameters indicates the potential for modelling electoral processes and wider applicability for complex social processes, avoiding dependence on "internal" variables.