Subtype classification and heterogeneous prognosis model construction in precision medicine

Common diseases including cancer are heterogeneous. It is important to discover disease subtypes and identify both shared and unique risk factors for different disease subtypes. The advent of high‐throughput technologies enriches the data to achieve this goal, if necessary statistical methods are developed. Existing methods can accommodate both heterogeneity identification and variable selection under parametric models, but for survival analysis, the commonly used Cox model is semiparametric. Although finite‐mixture Cox model has been proposed to address heterogeneity in survival analysis, variable selection has not been incorporated into such semiparametric models. Using regularization regression, we propose a variable selection method for the finite‐mixture Cox model and select important, subtype‐specific risk factors from high‐dimensional predictors. Our estimators have oracle properties with proper choices of penalty parameters under the regularization regression. An expectation–maximization algorithm is developed for numerical calculation. Simulations demonstrate that our proposed method performs well in revealing the heterogeneity and selecting important risk factors for each subtype, and its performance is compared to alternatives with other regularizers. Finally, we apply our method to analyze a gene expression dataset for ovarian cancer DNA repair pathways. Based on our selected risk factors, the prognosis model accounting for heterogeneity consistently improves the prediction for the survival probability in both training and test datasets.