Large-N analysis of critical behavior and effective potential in an O(N) model with octic interaction in fractional dimension d = 8/3

We analyze the O(N) theory near the upper critical dimension dc= 8/3, where the octic interaction (ϕ⋅ϕ)4 is marginal. For d=dc−2ε we find a Wilson–Fisher–type non-Gaussian infrared fixed point associated with a tetracritical O(N) universality class and extract its exponents. At d = 8/3 the interacting fixed point merges with Gaussian, leading to mean-field scaling. Using a large-N expansion and a modified minimal subtraction scheme, we compute the effective potential and renormalization group functions up to next-to-next-to-leading order (NNLO). The renormalized potential remains bounded from below without constraints on the coupling constant, in contrast to similar models in three dimensions. At the identified nontrivial fixed points, we determine the critical exponents of the tetracritical O(N) universality class. Our results extend the Wilson–Fisher framework to marginal higher-order interactions in non-integer dimensions, with relevance to systems exhibiting fractal or disordered structure.