Degenerate Hopf and homoclinic bifurcations in a prey–predator model with Allee-inducing escape ability of prey

In this article, we developed and analyzed a two-dimensional predator–prey model that incorporated prey escape mechanisms, which diminished the predator’s capture efficiency, included spatial habitat complexity effects, and accounted for predator mortality trade-offs that intensified with increased foraging effort. A classical Holling type II functional response is used to describe predator–prey interactions and then modified it to integrate the influence of habitat complexity and prey escape strategies. We examined the non-negativity and boundedness of solutions of the system and investigated the stability properties of equilibrium points. Our analysis revealed the presence of various bifurcations including Bautin (generalized Hopf), Bogdanov–Takens (BT), and generalized homoclinic bifurcations. The system demonstrates multi-stability in various regions of the biparametric plane. Numerical simulations have been performed to validate the analytical results obtained in this study. Additionally, sensitivity analysis was carried out to determine the influence of parameter variations on the behavior of the system and the persistence of both species. Incorporation of prey’s escape ability induces the Allee effect on the predator population. The primary objective of our proposed study is to achieve a balanced predator–prey interaction within a complex environment and the escape behavior of the prey.