Transformer is a neural network architecture that has fundamentally changed the approach to Artificial Intelligence. Transformer was first introduced in the seminal paper "Attention is All You Need" in 2017 and has since become the go-to architecture for deep learning models, powering text-generative models like OpenAI's GPT, Meta's Llama, and Google's Gemini. Beyond text, Transformer is also applied in audio generation, image recognition, protein structure prediction, and even game playing, demonstrating its versatility across numerous domains.
\nFundamentally, text-generative Transformer models operate on the principle of next-token prediction: given a text prompt from the user, what is the most probable next token (a word or part of a word) that will follow this input? The core innovation and power of Transformers lie in their use of self-attention mechanism, which allows them to process entire sequences and capture long-range dependencies more effectively than previous architectures.
\nGPT-2 family of models are prominent examples of text-generative Transformers. Transformer Explainer is powered by the GPT-2 (small) model which has 124 million parameters. While it is not the latest or most powerful Transformer model, it shares many of the same architectural components and principles found in the current state-of-the-art models making it an ideal starting point for understanding the basics.
\nEvery text-generative Transformer consists of these three key components:
\nTo convert a prompt into embedding, we need to (1) tokenize the input, (2) obtain token embeddings, (3) add positional information, and finally (4) add up token and position encodings to get the final embedding.
\nTokenization is the process of breaking down the input text into smaller, more manageable pieces called tokens. These tokens can be a word or a subword. The full vocabulary of tokens is decided before training the model: GPT-2's vocabulary has 50,257 unique tokens.
\nGPT-2 (small) represents each token in the vocabulary as a 768-dimensional vector. These embedding vectors are stored in a matrix of shape (50,257, 768), containing approximately 39 million parameters.
\nThe Embedding layer also encodes information about each token's position in the input prompt. GPT-2 trains its own positional encoding matrix from scratch.
\nFinally, we sum the token and positional encodings to get the final embedding representation. This combined representation captures both the semantic meaning of the tokens and their position in the input sequence.
\nThe core of the Transformer's processing lies in the Transformer block, which comprises multi-head self-attention and a Multi-Layer Perceptron layer. The GPT-2 (small) model consists of 12 such blocks.
\nThe self-attention mechanism enables the model to capture relationships among tokens in a sequence. Multiple attention heads allow the model to consider these relationships from different perspectives.
\nEach token's embedding vector is transformed into three vectors: Query (Q), Key (K), and Value (V). A useful analogy: Query is the search text; Key is the title of each result; Value is the content of matched pages.
\nQuery, Key, and Value vectors are split into multiple heads—in GPT-2 (small)'s case, into 12 heads. Masked self-attention prevents access to future tokens. Softmax converts scores into probabilities. Head outputs are concatenated and linearly projected.
\nAfter multi-head self-attention, concatenated outputs pass through an MLP: a first linear layer expands dimensionality four-fold from 768 to 3072 with GELU, then a second linear layer compresses back to 768.
\nAfter all Transformer blocks, a final linear layer projects into a 50,257-dimensional logit space over the vocabulary; softmax yields next-token probabilities. Temperature, top-k, and top-p sampling control determinism vs. diversity.
\nLayer Normalization, Dropout, and Residual Connections stabilize training, prevent overfitting, and mitigate vanishing gradients.
\nTransformer Explainer runs a live GPT-2 (small) model in the browser (nanoGPT → ONNX Runtime), with a Svelte + D3.js interface for attention maps, temperature, and sampling controls.
\nCreated by Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Jay Wang, Seongmin Lee, Benjamin Hoover, and Polo Chau at the Georgia Institute of Technology.
","headings":[{"level":1,"text":"What is a Transformer?","id":"what-is-a-transformer"},{"level":1,"text":"Transformer Architecture","id":"transformer-architecture"},{"level":2,"text":"Embedding","id":"embedding"},{"level":3,"text":"Step 1: Tokenization","id":"step-1-tokenization"},{"level":3,"text":"Step 2. Token Embedding","id":"step-2-token-embedding"},{"level":3,"text":"Step 3. Positional Encoding","id":"step-3-positional-encoding"},{"level":3,"text":"Step 4. Final Embedding","id":"step-4-final-embedding"},{"level":2,"text":"Transformer Block","id":"transformer-block"},{"level":3,"text":"Multi-Head Self-Attention","id":"multi-head-self-attention"},{"level":3,"text":"MLP: Multi-Layer Perceptron","id":"mlp-multi-layer-perceptron"},{"level":2,"text":"Output Probabilities","id":"output-probabilities"},{"level":2,"text":"Auxiliary Architectural Features","id":"auxiliary-architectural-features"},{"level":2,"text":"Interactive Features","id":"interactive-features"},{"level":2,"text":"Who developed the Transformer Explainer?","id":"who-developed-the-transformer-explainer"}]}}