Multimodal CustOmics: A unified and interpretable multi-task deep learning framework for multimodal integrative data analysis in oncology

Characterizing cancer presents a delicate challenge as it involves deciphering complex biological interactions within the tumor’s microenvironment. Clinical trials often provide histology images and molecular profiling of tumors, which can help understand these interactions. Despite recent advances in representing multimodal data for weakly supervised tasks in the medical domain, achieving a coherent and interpretable fusion of whole slide images and multi-omics data is still a challenge. Each modality operates at distinct biological levels, introducing substantial correlations between and within data sources. In response to these challenges, we propose a novel deep-learning-based approach designed to represent multi-omics & histopathology data for precision medicine in a readily interpretable manner. While our approach demonstrates superior performance compared to state-of-the-art methods across multiple test cases, it also deals with incomplete and missing data in a robust manner. It extracts various scores characterizing the activity of each modality and their interactions at the pathway and gene levels. The strength of our method lies in its capacity to unravel pathway activation through multimodal relationships and to extend enrichment analysis to spatial data for supervised tasks. We showcase its predictive capacity and interpretation scores by extensively exploring multiple TCGA datasets and validation cohorts. The method opens new perspectives in understanding the complex relationships between multimodal pathological genomic data in different cancer types and is publicly available on Github.Author summary: Cancer diagnosis and treatment require a deep understanding of the intricate biological interactions within tumors. In this study, we developed Multimodal CustOmics, a deep-learning framework designed to integrate histopathology images and multi-omics data, providing a comprehensive approach to cancer characterization. Multimodal CustOmics excels in predictive performance for both classification and survival analysis, outperforming current state of the art methods. Beyond its predictive capabilities, Multimodal CustOmics offers multi-level interpretability, revealing critical insights into genes, pathways, and spatial interactions. This interpretability is essential for clinicians and researchers to understand the underlying mechanisms of cancer progression and treatment responses. Our model’s robustness to missing data ensures its feasibility in real-world clinical scenarios. By leveraging diverse datasets from TCGA, Multimodal CustOmics has shown its potential to offer new avenues in precision medicine, and highlights the importance of integrating diverse biological data to achieve a holistic understanding.