Omnireg-gpt: a high-efficiency foundation model for comprehensive genomic sequence understanding

The human genome contains a sophisticated array of elements that regulate gene activity and organismal functions. Developing a large window foundation model capable of efficiently processing long sequence inputs is essential yet challenging for decoding the multi-layered and complex landscape of the cis-regulatory elements. Here, we introduce OmniReg-GPT, a generative foundation model designed for the low-resource pretraining of long genomic sequences by optimized attention mechanism. During pretraining, OmniReg-GPT captures the complete distribution of regulatory elements across nucleotide to megabase scales with efficient training speed and memory usage. We demonstrate exceptional performance in downstream regulotary applications spanning the entire spectrum of genomic scales, including various cis-regulatory elements identification, context dependent gene expression prediction, single-cell chromatin accessibility analysis, and 3D chromatin contact modeling. As a generative model, OmniReg-GPT also holds the potential to generate candidate cell-type-specific enhancers through prompt engineering. Overall, OmniReg-GPT extends the boundaries of foundation models in the genomic field, and provides a valuable pretraining model resource which can be extensively applied for genomic researches.