Cross-Modal Attention Fusion Based Generative Adversarial Network for Text-to-Image Synthesis

The synthesis of images from attribute descriptors is an emerging and intricate domain within the realm of computer vision, which has various application potentials in public security and multimedia. Existing attribute vector-to-face (V2F) synthesis methods mainly generate faces based on attribute label vectors that lack rich semantic feature information, which leads to low-quality generated face images. To surmount this limitation, we advocate attribute word-to-face (W2F) synthesis, leveraging sequences of attribute words rich in semantic content. A novel Cross-Modal Attention Fusion Generative Adversarial Network (CMAFGAN) is proposed to generate faces from facial attribute words. CMAFGAN stands out due to its incorporation of two innovative components, CMAF and WFT, which are proposed to explore the correlation between image features and the corresponding attribute word features. Experimental results on the CelebA and LFW datasets demonstrate that our CMAFGAN achieves state-of-the-art performance, effectively improving the quality of the synthesised faces. In particular, the consistency between the predicted images and input attribute words (R-precision) on the CelebA and LFW datasets achieved 61.24% and 64.46% respectively, representing a substantial improvement over prior techniques. Moreover, CMAFGAN achieves comparable or better performance than the current best methods on text-to-image synthesis (R-precision 83.41% on caltech-ucsd birds-200-2011, CUB). Additionally, we explore the application of CMAFGAN for X-ray image synthesis from textual descriptions, yielding finely detailed images that exhibit high fidelity to the ground-truth.