On structure testing for component covariance matrices of a high dimensional mixture

By studying the family of p‐dimensional scale mixtures, the paper shows for the first time a non‐trivial example where the eigenvalue distribution of the corresponding sample covariance matrix does not converge to the celebrated Marčenko–Pastur law. A different and new limit is found and characterized. The reasons for failure of the Marčenko–Pastur limit in this situation are found to be a strong dependence between the p‐co‐ordinates of the mixture. Next, we address the problem of testing whether the mixture has a spherical covariance matrix. To analyse the traditional John's‐type test we establish a novel and general central limit theorem for linear statistics of eigenvalues of the sample covariance matrix. It is shown that John's test and its recent high dimensional extensions both fail for high dimensional mixtures, precisely because of the different spectral limit above. As a remedy, a new test procedure is constructed afterwards for the sphericity hypothesis. This test is then applied to identify the covariance structure in model‐based clustering. It is shown that the test has much higher power than the widely used integrated classification likelihood and Bayesian information criteria in detecting non‐spherical component covariance matrices of a high dimensional mixture.