How a transformer picks the next word

This is a real model with only 6,400 parameters, trained on 41 short sentences about AI. Everything below is computed live in your browser.

words→numbers→attention→MLP→× 2 layers→probabilities

Step 1

Tokens

The text is split into tokens. Here each word is one token with an ID from a vocabulary of 60 words.

Step 2

Embeddings + position

Each token becomes 16 numbers (an embedding) describing its meaning. A position vector is added so the model knows word order.

negative positive

Step 3

Query · Key · Value

Every token makes a Query ("what am I looking for?"), a Key ("what do I contain?") and a Value ("what I'll share"). Columns 1–8 belong to head 1, columns 9–16 to head 2.

Step 4

Attention

Each word compares its Query with the Keys of earlier words to decide how much to look at each one. Rows are the word looking and columns are the word looked at. No peeking ahead. Pick a head, or click a row.

Step 5

MLP (feed-forward)

Each token then goes through a small neural network on its own (16 → 32 → 16). The bright cells are "neurons" that fired. Steps 3–5 repeat for each of the 2 layers.

Step 6

Next-word probabilities

The last token's vector becomes a score for every word, and softmax turns the scores into probabilities. Temperature divides the scores first: lower sharpens the odds, higher flattens them.