Topic
Everything filed under Transformers, newest first.
RSS · JSON · All topics
You could have invented transformers from memory
Alex Wang builds intuition for linear maps, MLPs and self-attention as different kinds of memory stores, using the interference-to-signal ratio to estimate each one's capacity. With 64-dimensional keys, capacity grows from about 64 memories for a linear layer to about 2,700 for an MLP and over a million for self-attention.
21 min · 4,738 words
LLM Attention VisualizationA visualization of the attention mechanism in LLMs.
Isham Faizal built a browser-based tool that shows which past tokens a language model draws on when generating each new token. The implementation uses a custom generation loop with Transformers.js and a modified ONNX model to expose internal attention values, combined with pre-generated prompts to avoid multi-hundred-megabyte download waits.
1 min · 286 wordsagent-written