How Does the Transformer Attention Mechanism Work? (Queries, Keys & Values Explained)

The Bottleneck of RNNs: The Road to "Attention Is All You Need" (2017)

Prior to 2017, the state of the art in Natural Language Processing (NLP) relied on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks [1,2]. These architectures processed text sequentially, token by token from left to right [1,2].

Sequential processing created two fatal bottlenecks [1,2]:

1. Information Vanishing: As sentences grew longer (over 50 to 100 words), earlier contextual information was compressed into a fixed-size hidden state vector, causing the model to forget distant words [1,2].

2. No GPU Parallelization: Because step $t$ depended strictly on the hidden state of step $t-1$, training could not be parallelized across thousands of GPU cores [1,3].

In June 2017, a team of eight Google Brain and Google Research scientists (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin) published the landmark paper: Attention Is All You Need [1,3]. They eliminated recurrence entirely, introducing the Transformer—an architecture where every word attends directly to every other word simultaneously in parallel [1,3].

"The 2017 Transformer eliminated sequential recurrence entirely, allowing neural networks to process all words in parallel with direct pairwise attention."

The Core Equation: Scaled Dot-Product Attention

At the heart of the Transformer is Scaled Dot-Product Attention, defined by the fundamental mathematical equation [1,3,4]:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

For any input token embedding $X$, the network learns three distinct weight projection matrices ($W_Q, W_K, W_V$) to generate three vectors [1,3,4]:

1. Query ($Q$): "What am I looking for?" (e.g., the pronoun "it" searching for its antecedent) [1,4].

2. Key ($K$): "What is my identity and content?" (every word advertising its semantic attributes) [1,4].

3. Value ($V$): "What informative content do I provide if matched?" [1,4].

"In self-attention, Query matrices ask what information is needed, Key matrices broadcast identity, and Value matrices deliver content."

Why Scale by $\sqrt{d_k}$? Preventing Gradient Vanishing in Softmax

The dot product $Q \cdot K^T$ computes a raw cosine-like similarity score between every pair of tokens in the sequence [1,3,5]. For large vector dimensions (such as $d_k = 64$ or $128$), dot products grow large in magnitude, pushing the softmax function into regions with extremely small gradients (gradient saturation) where learning stalls [1,3,5].

Dividing by the scaling factor $\sqrt{d_k}$ normalizes the variance back to 1.0, ensuring smooth gradients during backpropagation [1,3,5]. The resulting softmax probabilities represent an Attention Map—a probability distribution indicating how much weight token $A$ should assign to token $B$ when updating its representation [1,4,5].

Multi-Head Attention & The Foundation of Modern LLMs

Rather than computing a single attention function, Transformers use Multi-Head Attention [1,3,6]. By splitting the queries, keys, and values into $h$ independent subspaces (typically 8, 32, or 128 heads), each head specializes in different linguistic relationships in parallel [1,3,6]:

- Head 1 might track direct syntactic subject-verb agreements ("The cat ... sits").

- Head 2 might resolve pronoun coreferences ("it" $\to$ "the bank").

- Head 3 might track long-range temporal markers and causal relationships across paragraphs [1,6,7].

Concatenating the outputs of all attention heads with Feed-Forward Networks (FFNs) and residual layer normalization enables the massive emergent reasoning and in-context learning capabilities of modern frontier models (GPT-4, Claude 3.5, Gemini 1.5) [1,3,7].