Interplay between preys’ anti-predator behavior and predators’ cooperative hunting: A mathematical approach

In ecosystems, hunting cooperation of the predator will intensify the survival pressure faced by the prey, and then promote anti-predator behavior in the prey population. Therefore, it is of biological significance to study the prey’s anti-predator behavior and the predator’s hunting cooperation simultaneously. In this paper, a mathematical model that encompasses both aspects is developed. Through rigorous and comprehensive analysis of equilibria and bifurcation phenomena, this study reveals how these two behaviors influence population dynamics and ecosystem stability. In detail, as the hunting cooperation parameter a increases, the density of the prey population decreases. When a is extremely large or small, it will hinder the persistence of predator and prey populations, and only moderate hunting cooperation can maintain the ecosystem balance. Furthermore, moderate anti-predator behavior of the prey can suppress predators’ hunting efficiency, while once the anti-predator behavior exceeds a critical threshold, it will disrupt the coexistence of predator and prey populations. The findings in this research can be viewed as an improvement and a complement to those with anti-predator behavior or hunting cooperation.