scPEDSSC: proximity enhanced deep sparse subspace clustering method for scRNA-seq data

It is a significant step for single cell analysis to identify cell types through clustering single-cell RNA sequencing (scRNA-seq) data. However, great challenges still remain due to the inherent high-dimensionality, noise, and sparsity of scRNA-seq data. In this study, scPEDSSC, a deep sparse subspace clustering method based on proximity enhancement, is put forward. The self-expression matrix (SEM), learned from the deep auto-encoder with two part generalized gamma (TPGG) distribution, are adopted to generate the similarity matrix along with its second power. Compared with eight state-of-the-art single-cell clustering methods on twelve real biological datasets, the proposed method scPEDSSC can achieve superior performance in most datasets, which has been verified through a number of experiments.Author summary: The rapid advancement of single-cell RNA sequencing technologies has thrown a new light on studying complex biological phenomena. A crucial step in the single-cell transcriptome analysis is to group cells which belong to the same cell type with gene expression data, i.e., clustering a noisy, sparse and high dimensional dataset with enormously fewer cells than the number of genes. In order to address the above problems, we propose a deep sparse subspace clustering method based on proximity enhancement. The raw sequencing data are first preprocessed by four different similarities and the corresponding Laplace scores to initially reduce their dimensionality. Afterwards, the self-expression matrix (SEM), learned from the deep auto-encoder with two part generalized gamma (TPGG) distribution, are adopted to generate the similarity matrix along with its second power. The clustering results are finally obtained using spectral clustering. Experimental comparisons with eight state-of-the-art methods on multiple datasets demonstrate the effectiveness and reliability of method scPEDSSC in clustering scRNA-seq data.