Attention Mechanism

The attention mechanism is the core innovation behind the transformer architecture and modern AI. It allows a model to assign different levels of importance (attention weights) to different parts of the input when processing each position in the sequence. Instead of compressing an entire input into a single fixed-size representation, attention lets the model selectively focus on the most relevant information for each step of its computation.

Self-attention, the specific form used in transformers, computes attention scores between every pair of positions in the input sequence. Each token generates three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what information do I provide?). The attention score between two tokens is computed by comparing the query of one with the key of the other, and these scores determine how much each token's value contributes to the representation of every other token. Multi-head attention runs this process multiple times in parallel with different learned projections, capturing different types of relationships.

Attention is what allows LLMs to understand context, resolve references, and maintain coherence across long passages. When the model processes the word "it" in a sentence, attention weights light up on the noun "it" refers to, even if they are many words apart. This ability to capture long-range dependencies without the information bottleneck of sequential processing is why transformers dramatically outperform earlier architectures on language tasks.