ST-CellSeg: Cell segmentation for imaging-based spatial transcriptomics using multi-scale manifold learning

Spatial transcriptomics has gained popularity over the past decade due to its ability to evaluate transcriptome data while preserving spatial information. Cell segmentation is a crucial step in spatial transcriptomic analysis, as it enables the avoidance of unpredictable tissue disentanglement steps. Although high-quality cell segmentation algorithms can aid in the extraction of valuable data, traditional methods are frequently non-spatial, do not account for spatial information efficiently, and perform poorly when confronted with the problem of spatial transcriptome cell segmentation with varying shapes. In this study, we propose ST-CellSeg, an image-based machine learning method for spatial transcriptomics that uses manifold for cell segmentation and is novel in its consideration of multi-scale information. We first construct a fully connected graph which acts as a spatial transcriptomic manifold. Using multi-scale data, we then determine the low-dimensional spatial probability distribution representation for cell segmentation. Using the adjusted Rand index (ARI), normalized mutual information (NMI), and Silhouette coefficient (SC) as model performance measures, the proposed algorithm significantly outperforms baseline models in selected datasets and is efficient in computational complexity.Author summary: Spatial transcriptomics data is a type of biological data that describes gene expression patterns in the context of tissue or cell spatial arrangement. Traditional transcriptomics studies the gene expression of a group of cells or a tissue sample as a whole, revealing which genes are active or inactive in that sample. Spatial transcriptomics, on the other hand, is a recent technology that can maintain the spatial information of where these genes are expressed inside the tissue. These methods provide a more accurate description of tissue and cell subcellular architecture, allowing for a better understanding of physical and biochemical interactions between cells. Precise cell identification is critical because it can aid in the discovery of unusual cell types, particularly in cancer research. Traditional clustering approaches, on the other hand, frequently fail to account for spatial information. The issue in bioinformatics is thus to diversify cell segmentation approaches in spatial transcriptomic analysis. To that purpose, we develop a cell segmentation technique for spatial transcriptomic data that uses distance metrics to better define the spatial transcriptomics distribution. The experimental results reveal that this algorithm outperforms the popular cell segmentation algorithms and performs faster under the same conditions.