An Explainability-Driven Framework for Interpretable Cross-Modal Image-Text Retrieval Using CLIP

Large vision-language models have been used to create cross-modal retrieval systems, including CLIP, that have achieved large performance improvements but often act as black boxes, which makes it more difficult to use these models and apply them in more critical areas. This non-disclosure is a great impediment to the responsible implementation of such systems in high stakes applications. As a measure to counter this shortcoming, we suggest an explainability-based design with an embedded post-hoc interpretation modules as part of the CLIP retrieval pipeline. The framework provides sensible, dual- mode accounts of bidirectional retrieval tasks; to begin with, it produces visual heatmaps that both outline the regions in the image that have the strongest impact on a retrieval decision and to a second end, it deactivates word-level attribution to quantify the relative significance of textual tokens in the query or caption. As we will discuss later, with our implementation and subsequent evaluation of our system on Flickr8k one can see that we provide these interpretable insights whilst maintaining, and in fact slightly increasing the baseline retrieval accuracy of the vanilla CLIP model. The empirical evidence confirms that the incorporation of interpretability layers is not accompanied by the trade-off in terms of performance. Together, this work confirms that principled explainability mechanisms should be augmented to multimodal retrieval systems in order to foster trustful, responsible AI solutions. Based on the increased transparency, the approach prepares the foundation of more solid and trustworthy human-AI cooperation.