Solution sets of three sparse optimization problems for multivariate regression

In multivariate regression analysis, a coefficient matrix is used to relate multiple response variables to regressor variables in a noisy linear system from given data. Optimization is a natural approach to find such coefficient matrix with few nonzero rows. However, the relationship of most group sparse optimization models with cardinality penalty or cardinality constraints is not clear. In this paper, we give a comprehensive description of the relationship between three widely used group sparse optimization problems with cardinality terms: (i) the number of nonzero rows is minimized subject to an error tolerance for regression; (ii) the error for regression is minimized subject to a row cardinality constraint; (iii) the sum of the number of nonzero rows and error for regression is minimized. The first two problems have convex constraints and cardinality constraints respectively, while the third one is an unconstrained optimization problem with a cardinality penalty. We provide sufficient conditions under which the three optimization problems have the same global minimizers. Moreover, we analyze the relationship of stationary points and local minimizers of the three problems. Finally, we use two examples to illustrate our theoretical results for finding solutions of constrained optimization problems involving cardinality terms by unconstrained optimization problems with penalty functions.