The impact of higher-order dynamics with intraspecific competition in replicator dynamics on species stability

Higher-order interactions, as crucial factors in natural ecosystems, can be prominently observable across diverse ecological systems. Previous studies have predominantly focused on pairwise interspecific interactions. In this work, we investigate how higher-order interactions (HOI) and intraspecific competition jointly influence ecosystem stability in a rock–paper–scissors (RPS) model by incorporating intraspecific competition factors into a replicator dynamics framework that integrates both pairwise and higher-order interactions. Our findings reveal that the system undergoes a Hopf bifurcation within specific parameter ranges, generating a subcritical limit cycle manifested as periodic oscillations in species abundance. Conversely, the system traverses the Hopf bifurcation when critical parameters exceed threshold values, driving all species toward extinction via boundary attractors. Notably, the strength of intraspecific competition significantly modulates system stability, and under specific combinations of intensities influences system dynamics and the properties of limit cycles. Our study highlights the importance of incorporating multidimensional intraspecific competition into game-theoretic models with higher-order interactions to understand biodiversity maintenance and ecosystem stability. We hope this work may provide novel theoretical insights into the multidimensional effects of intra- and inter-species interactions in complex ecological networks.