Coevolutionary dynamics of higher-order interactions and strategic behavior in social dilemmas

The interplay between collective behavioral strategies and dynamic interaction topologies is a defining feature of complex social systems. While classical evolutionary game theory predominantly assumes static environments, we introduce a continuous coevolutionary framework where the structural order of interactions — shifting dynamically between dyadic links and higher-order simplexes — evolves endogenously as a macroscopic topological order parameter. Driven by a bidirectional eco-evolutionary feedback loop, cooperators actively synthesize higher-order structures, while defectors drive structural degradation. By coupling replicator dynamics with logistic structural evolution, we reveal that the system’s macroscopic fate is governed by the interplay between the Higher-Order Exploitation Gap (Δmix), the relative topological plasticity (θ), and underlying pairwise dilemma type (characterized by T and S). We demonstrate that topological feedback acts as a profound, nonlinear evolutionary driver: In the Prisoner’s Dilemma, it enables the nucleation of a bistable “structural niche,” allowing cooperation to escape the tragedy of the commons. In the Stag Hunt, the topology acts as a nonlinear amplifier of initial conditions, inducing extreme hysteresis and irreversible “all-or-nothing” phase transitions. Conversely, in the Harmony and Chicken games, severe multi-body exploitation (Δmix>0) triggers symmetry breaking, generating paradoxical interior traps and thermodynamic mixed states that dynamically prevent Pareto-optimal convergence. These findings establish that adaptive topologies act as elastic, self-regulating mechanisms, offering a unified statistical-physics perspective on the co-emergence of structural complexity and social cooperation.