RareCapsNet: An explainable capsule network enables robust discovery of rare cell populations from large-scale single-cell transcriptomics

Large-scale single-cell RNA sequencing has created new opportunities for computational identification of rare cell populations from complex transcriptomic data. Improvements in sequencing throughput now enable the profiling of tens of thousands of cells, increasing the likelihood of capturing rare cell types. We develop RareCapsNet, a rare cell identification technique through capsule network in large single cell RNA-seq data. RareCapsNet aiming to leverage the landmark advantages of capsule networks in single cell domain, by identifying rare, poorly represented, or putative rare cell populations through markers genes explained from human-mind-friendly interpretation of lower-level (primary) capsules. We demonstrate the explainability of capsule network for identifying capsule-associated markers that are act as signature of certain cell population of rare type. A comprehensive evaluation in simulated and real life single cell data demonstrate the efficacy of RareCapsNet for finding out rare population in large scRNA-seq data. RareCapsNet outperforms the other state-of-the-art not only in specificity and selectivity for identifying rare cell types, it can also successfully extract transcriptomic signature of the cell population. We demonstrate RareCapsNet to the dataset of multiple batch, where the model can store the knowledge of one batch which can be transferred to find out rare cells of other batch without training the model. Availability and Implementation: RareCapsNet is available at: https://github.com/sumantaray/RareCapsNet.Author summary: Single-cell RNA sequencing allows researchers to study thousands of individual cells and understand how different cell types contribute to health and disease. However, some biologically important cells occur in very small numbers. These rare cells may include immune subtypes, cancer-related cells, progenitor cells, or specialized neuronal populations. Because they are poorly represented, standard computational methods may miss them or fail to explain which genes make them distinct. We developed RareCapsNet, a capsule-network-based method for detecting rare cell populations in large single-cell transcriptomic datasets. The model learns relationships between gene-expression patterns and cell types through capsule routing, which helps it identify rare or underrepresented cells more accurately. Importantly, RareCapsNet also provides an interpretable way to connect predicted rare cell types with marker genes by analyzing capsule-coupling patterns. We tested RareCapsNet on simulated and real single-cell datasets, including immune and brain datasets, and compared it with existing rare-cell detection methods. Our results show that RareCapsNet can improve rare-cell identification while also suggesting biologically meaningful genes associated with these rare populations.