Self-attention vector output similarities reveal how machines pay attention

The self-attention mechanism has significantly advanced the field of natural language processing, facilitating the development of advanced language-learning machines. Although its utility is widely acknowledged, the precise mechanisms of self-attention underlying its advanced learning and the quantitative characterization of this learning process remains an open research question. This study introduces a new approach for quantifying information processing within the self-attention mechanism. The analysis conducted on the BERT-12 architecture reveals that, in the final layers, the attention map focuses on sentence separator tokens, suggesting a practical approach to text segmentation based on semantic features. Based on the vector space emerging from the self-attention heads, a context similarity matrix, measuring the scalar product between two token vectors was derived, revealing distinct context similarities between different token vector pairs within each head and layer. The findings demonstrated that different attention heads within an attention block focused on different linguistic characteristics, such as identifying token repetitions in a given text or recognizing a token of common appearance in the text and its surrounding context. This specialization is also reflected in the distribution of distances between token vectors with high context similarities as the architecture progresses. The initial attention layers exhibit substantially long-range context similarities; however, as the layers progress, a more short-range context similarity develops, culminating in a preference for attention heads to create strong context similarities within the same sentence. Finally, the behavior of individual heads was analyzed by examining the uniqueness of their most common tokens in their high context similarity elements. Each head tends to focus on a unique token from the text and builds similarity pairs centered around it. This methodology for quantifying the behavior of the vector space emerging from the self-attention layers offers a novel way to understand the dynamics of the self-attention mechanism with respect to its outputs.