ASATrans: Adaptive spatial aggregation transformer for cervical nuclei segmentation on rough edges

The main characteristic of cervical cytopathy is reflected in the edge shape of nuclei. Existing computer-aided diagnostic techniques can clearly segment individual nuclei, but cannot clearly segment the rough edges of adherent nucleus. Therefore, we propose an effective method (ASATrans) to accurately segment rough cervical nuclei edges by exploring adaptive spatial aggregation methods. ASATrans creates a Multi-Receptive Embedding Layer that samples patches using diverse-scale kernels. This approach provides cross-scale features to each embedding, preventing semantic corruption that might arise from mapping disparate patches to analogous underlying representations. Furthermore, we design Adaptive Pixel Adjustment Block by introducing a long-range dependency and adaptive spatial aggregation. This is achieved through the stratification of the spatial aggregation process into distinct groups. Each group is given an exclusive sampling volume and modulation scale, fostering a collaborative learning paradigm that combines local features and global dependencies. This collaborative approach to feature extraction achieves adaptability, mitigates interference from unnecessary pixels, and allows for better segmentation of edges in the nucleus. Extensive experiments on two cervical nuclei datasets (HRASPP Dataset, ISBI Dataset), demonstrating that our proposed ASATrans outperforms other state-of-the-art methods by a large margin.