Using, taming or avoiding the factor zoo? A double-shrinkage estimator for covariance matrices

Existing factor models struggle to model the covariance matrix for a large number of stocks and factors. Therefore, we introduce a new covariance matrix estimator that first shrinks the factor model coefficients and then applies nonlinear shrinkage to the residuals and factors. The estimator blends a regularized factor structure with conditional heteroskedasticity of residuals and factors and displays superior all-around performance against various competitors. We show that for the proposed double-shrinkage estimator, it is enough to use only the market factor or the most important latent factor(s). Thus there is no need for laboriously taking into account the factor zoo.