Concentration, existence of a ground state and multiplicity of solutions for a subcritical elliptic system via penalization method

We consider the system $$\begin{aligned} \left\{ \begin{array}{l} -\varepsilon ^{2} \text{ div }(a(x) \nabla u)+u=Q_{u}(u,v)~\quad \text {in } \mathbb {R}^N, \\ -\varepsilon ^{2} \text{ div }(b(x) \nabla v) +v=Q_{v}(u,v)~\quad \text {in } \mathbb {R}^N, \\ u,v \in H^{1}(\mathbb {R}^N),u(x),v(x)>0\,\,\quad \,\,\, \text {for each } x \in \mathbb {R}^N, \end{array} \right. \end{aligned}$$ - ε 2 div ( a ( x ) ∇ u ) + u = Q u ( u , v ) in R N , - ε 2 div ( b ( x ) ∇ v ) + v = Q v ( u , v ) in R N , u , v ∈ H 1 ( R N ) , u ( x ) , v ( x ) > 0 for each x ∈ R N , where $$\varepsilon >0$$ ε > 0 , a and b are positive continuous potentials and Q is a p-homogeneous function with subcritical growth. In the first place we show existence of a ground state solution for this system. After that, we show existence of multiple solutions involving the category theory and the topology of the sets of minima of the potentials a and b . Finally, we show a concentration result. More precisely, we show that at the maximum points of each solution, the potentials a and b converge to their points of minimum points when $$\varepsilon$$ ε converges to zero.