Vision-Language Pre-training from Synthetic Data

Recent advancements in Medical Vision-Language Pre-training (MedVLP) demonstrate significant potential, leveraging extensive datasets of medical images and accompanying reports to deliver impressive performance across a wide range of downstream tasks, including both visual-based challenges and those integrating vision and language. However, MedVLP systems require substantial datasets with matched image-text pairs, which are often challenging to procure due to their labor-intensive and costly nature. Additionally, real-world datasets frequently encounter issues such as imbalanced concepts, unpaired image-text samples, and corrupted images. Recent progress in deep generative models, notably from Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to Stable Diffusion (SD)-based models, has been significant. SD-based models, in particular, excel in conditional generation, a crucial capability for synthesizing medical images with high fidelity. Moreover, the generation of medical reports can be enhanced using language models, especially large language models (LLMs) such as Llama, utilizing conditional generation driven by extensive medical concept definitions sourced from clinical peer-reviewed databases. This article introduces the principal MedVLP methodologies, the role of generative models, and the techniques of conditional generation, alongside an exploration of various downstream tasks employed to assess the effectiveness of MedVLP.